A Behavioral Economic Reward Index Predicts Drinking Resolutions: Moderation Revisited and Compared With Other Outcomes

Acknowledgement: Mary J. Vignolo is now affiliated with the American College of Rheumatology Research and Education Foundation, Atlanta, GA. Andrew O. Westfall is now affiliated with the Department of Obstetrics and Gynecology, University of Alabama at Birmingham School of Medicine, and the Centre for Infectious Disease Research, Lusaka, Zambia.

This research was supported in part by Grants R01 AA008972 and K02 AA000209 from the National Institute on Alcohol Abuse and Alcoholism. The authors thank Paula D. Rippens, Bethany C. Black, and H. Russell Foushee for their contributions to the data collection phase of the research, and G. Alan Marlatt for his comments on an earlier version of this article. Portions of this research were presented at the Third Annual Workshop Mechanisms of Behavior Change in Behavioral Treatment held at the annual meeting of the Research Society on Alcoholism, Chicago, IL, July 2007.

Behavioral economic models of choice behavior have been widely applied to an analysis of substance misuse and other addictive behaviors in humans (e.g., Bickel & Marsch, 2001; Green & Kagel, 1996; Vuchinich & Tucker, 1996, 1998). Behavioral economics involves a merger of operant approaches to understanding choice behavior, particularly impulsive choice (Ainslie, 1975), with microeconomic models of consumer behavior (Rachlin, Battalio, Kagel, & Green, 1981). Both focus on how individuals allocate limited resources such as time, money, and behavior to obtain commodities available at different costs and over different delays, and strength of preference for a given commodity (e.g., drug use) is inferred from the relative resources or behavior allocated to obtain it (Premack, 1965; Rachlin, 1971). For example, the well-established matching law quantifies how humans and animals alike distribute or “match” relative response rates in proportion to the relative rates of reinforcement available from different activities (Herrnstein, 1970).

This approach is well suited to studying demand for drugs in relation to other commodities available in the natural environment (Vuchinich & Tucker, 1996, 1998). Behavioral economic models view substance misuse as a persistent preference for short-term rewards and a devaluation of larger, delayed rewards that support adaptive functioning. Research has consistently shown that preference for substance use decreases as constraints on access to the substance increase and as constraints on access to valued non-drug-related alternatives decrease (Vuchinich & Tucker, 1998). Moreover, persons with addictive behavior problems tend to devalue, or discount, delayed rewards more than normal controls (Bickel & Marsch, 2001). Control of their current behavior is less sensitive to delayed consequences, such as the adverse long-term effects of substance use.

As applied to attempts to resolve alcohol problems, these findings suggest that problem drinkers with greater sensitivity to longer term contingencies, even when drinking heavily, should have a better prognosis and that shifting control of behavior from shorter to longer term contingencies should promote resolution stability. Our earlier prospective studies of resolution attempts by community-dwelling treated and untreated problem drinkers supported this hypothesis (Tucker, Foushee, & Black, 2008; Tucker, Vuchinich, Black, & Rippens, 2006; Tucker, Vuchinich, & Rippens, 2002). Shortly after initiation of abstinence or problem-free moderation drinking, participants reported their monetary expenditures on alcoholic beverages and other commodities during the year before resolution onset using an expanded Timeline Followback (TLFB) interview (Sobell & Sobell, 1992). Establishing relative preference for alcohol through allocation of monetary resources is based on experimental work showing that the relative values of different, concurrently available commodities can be quantified by measuring choice among the commodities under varying constraints (Herrnstein, 1970; Premack, 1965). Many different activities are available in the natural environment, and monetary allocation offers a common metric to assess their relative reinforcement value (Vuchinich, Tucker, & Harllee, 1988).

Our main research focus was on studying successful natural recoveries achieved without treatment. Natural recovery samples typically include middle-income to upper income individuals (Sobell, Sobell, & Toneatto, 1992) who have complex, fixed, and recurring expenditures (e.g., mortgages, automatic payroll deductions) as well as considerable discretionary expenditures (e.g., for recreation, alcohol, voluntary savings). Because their preferences should be more readily expressed within discretionary, as opposed to fixed, spending patterns, the proportion of discretionary spending on alcoholic beverages was compared with money put into savings for future use, which was conceptualized as representing the value of rewards available over shorter and longer time horizons, respectively. Greater relative allocation to savings than to drinking, reflected in lower values on this Alcohol-Savings Discretionary Expenditure (ASDE) index, was viewed as indicating higher relative preferences for delayed rewards made possible by savings compared with more immediate rewards from drinking.

As hypothesized, problem drinkers who maintained stable resolutions had lower preresolution ASDE values than those who had unstable resolutions and relapsed at any point during the 1- to 2-year follow-ups (Tucker et al., 2002, 2006, 2008). The ASDE index had unique incremental utility in predicting stable versus unstable resolutions after controlling for established outcome predictors (e.g., problem severity, drinking practices), and it had predictive utility across intervention-naive and intervention-exposed resolution groups (Tucker et al., 2006). In addition to predicting long-term resolution stability, the ASDE index predicted drinking patterns during the early months of the postresolution period in a study that implemented Interactive Voice Response (IVR) self-monitoring with recently resolved, untreated problem drinkers (Tucker et al., 2008).

This research showed that contextually sensitive measures of the reward value of drinking in relation to other activities added unique information in an account of resolution outcomes. However, in these studies many more participants achieved stable abstinent than nonabstinent resolutions, so predictors of moderation apart from abstinence could not be investigated in the studies individually. A more extensive analysis with a larger sample that includes more participants who drank in a sustained nonproblem manner is needed to examine specific predictors of moderation. This issue has gained renewed importance as interventions continue to expand beyond abstinence-oriented treatments for alcohol-dependent persons, to include population-based public health interventions for the untreated majority with less severe problems for whom moderation is a more common and acceptable outcome (Tucker, 2003). Moderation outcomes are more common among untreated problem drinkers who quit on their own compared with the minority who seek treatment, partly because treatment seekers have more serious problems. Although early treatment research found moderation to be associated with lower problem severity, younger age, and stable life circumstances (reviewed by Miller & Munoz, 2005; Rosenberg, 2004), there have been few recent advances, with the exception that higher self-efficacy to resist drinking in high risk situations has been associated with moderation outcomes (Saladin & Santa Ana, 2004).

To obtain a sufficient sample to investigate stable moderation apart from other outcomes, we pooled the data from our three prior studies and conducted new analyses to evaluate the utility of the ASDE index in distinguishing stable nonabstinent resolutions from stable abstinent resolutions and unstable resolutions that involved problem drinking at some point over the 1- to 2-year follow-ups. Our interest in examining this issue in a re-analysis comes from early theorizing about the processes involved in moderation (Marlatt, 1985) and from preliminary findings in our IVR study (Tucker et al., 2008) that supported the theorizing. Over two decades ago, Marlatt (1985, pp. 329–344) raised the interesting but still unstudied hypothesis that abstinence and relapse are opposite ends of the same dynamic behavioral regulation process, reflecting over- and undercontrol of the daily act of drinking, respectively. Moderation was thought to involve a different regulation process that depends on “lifestyle balance” and repetitive choices to drink well within the boundaries of extreme restraint or loss-of-control drinking. To the extent that the ASDE index is a functional measure of preference for alcohol in relation to delayed rewards made possible by savings, one would expect successful moderate drinkers to organize their behavioral allocation (tracked via financial expenditures) over longer intervals compared with those who relapse or abstain. Framing Marlatt's (1985) “differential regulation” hypothesis within behavioral economic theory, lower ASDE values, reflecting more balanced monetary allocations between short-term and longer term objectives (i.e., drinking and saving for the future), should predict moderation compared with other outcomes.

The pooled data set included 30 drinkers with moderation outcomes, which was sufficient to evaluate the hypothesis that stable resolutions involving some moderation drinking over 1 to 2 years would be predicted by lower preresolution ASDE values compared with other outcomes. After this primary behavioral economic hypothesis was evaluated, established moderation predictors assessed at baseline, including problem severity, alcohol dependence, and self-efficacy, were included with the ASDE index in multinomial logistic regression models to determine if the index had unique incremental predictive utility in distinguishing outcomes among participants who resumed drinking (relapse or moderation) and among those who remained resolved (abstinence or moderation). A series of multivariable models were used to subject the ASDE index to a rigorous systematic evaluation after controlling for multiple covariates and to maintain favorable events-to-predictor ratios in each model (Peduzzi, Concato, Kemper, Holford, & Feinstein, 1996) within the limits set by the number of moderation cases in the pooled data set. We hypothesized that the ASDE index would provide significant incremental predictive utility over predictors suggested by prior research and would be particularly sensitive for distinguishing moderated and relapsed outcomes—that is, the differential regulation processes theorized by Marlatt (1985) and assessed by our behavioral economic index should be most apparent among participants who engaged in some postresolution drinking.

Sample Selection and Characteristics

Participants were recruited from the community via media advertisements in metropolitan areas in Alabama, Florida, Georgia, Mississippi, and Tennessee. The advertisements asked for research volunteers who had recently overcome a drinking problem with or without treatment. Respondents to the advertisements called a toll-free number, received a description of the research, and were screened with the Michigan Alcoholism Screening Test (MAST; Selzer, 1971), Alcohol Dependence Scale (ADS; Skinner & Horn, 1984), and Drinking Problems Scale (DPS; Cahalan, 1970). Eligible participants were scheduled for interviews in a place convenient for them. All studies were conducted in compliance with university institutional review board and American Psychological Association ethical standards for research with humans. Participants were informed that the research was covered by a confidentiality shield issued by the U.S. Department of Health and Human Services.

Eligibility criteria included a minimum 5-year drinking problem history (M = 16.70 years, SD = 9.31), no current other drug misuse (except nicotine), and recent cessation of problem drinking (M = 3.93 months resolved, SD = 1.78). Resolution onset was defined as the most recent date that participants began abstaining or drinking in a nonproblem manner without further heavy drinking. At all assessments, moderation was determined using criteria associated with low health risks related to drinking (Sobell et al., 1992): (a) <55 g (70 ml) of 190-proof ethanol consumed per drinking day; (b) no dependence symptoms (as assessed by the ADS); and (c) no alcohol-related negative health, psychosocial, vocational, financial, or legal consequences (as assessed by the DPS). These criteria are consistent with other drinking guidelines (e.g., National Institute on Alcohol Abuse and Alcoholism [NIAAA], 2005; World Health Organization, 2000) that generally set upper limits at ≤4 drinks/day for men and ≤3 drinks/day for women.

All of the studies included untreated problem drinkers who had initiated natural resolutions, and Tucker et al. (2006) also included a group that had received alcohol treatment from a qualified provider or attended more than two Alcoholics Anonymous meetings within about 12 months of resolution onset. Two studies (Tucker et al., 2002, 2006) required an initial resolution of 2 to 6 months and had a 2-year follow-up; the third study that involved IVR self-monitoring (Tucker et al., 2008) required a shorter initial resolution of 1 to 3 months and had a 1-year follow-up. Otherwise, the studies used identical selection criteria and follow-up procedures. Summed across studies, 205 (81.03%) of the 253 initially enrolled participants completed the 1-year follow-up required for inclusion in the pooled sample; 202 provided useable income and expenditure data and were included in the data analyses. Attrition was due to participant withdrawal or lost contact (42), significant discrepancies between participant and collateral reports of drinking (5), or death (1).

Although not an inclusion criterion, all participants met third- (Tucker et al., 2002) or fourth- (Tucker et al., 2006, 2008) edition Diagnostic and Statistical Manual of Mental Disorders criteria for alcohol dependence (American Psychiatric Association, 1987, 1994). As shown in Table 1, ADS scores fell in the moderate to low substantial dependence range (Skinner & Horn, 1984). Consistent with research showing that seeking help is associated with more severe problems, ADS, MAST, and DPS scores were relatively higher in the one study that included participants with a help-seeking history (Tucker et al., 2006). Otherwise, no differences were found across studies in preresolution drinking practices, postresolution outcomes, demographic characteristics, and preresolution income and expenditures on alcoholic beverages and money put into savings.

Sample Characteristics at Initial Assessment as a Function of Resolution Status at Follow-Up

Because our goal was to predict stable moderation apart from other outcomes, we classified participants conservatively into mutually exclusive groups based on drinking practices and problems over the entire follow-up. Those who abstained or drank moderately without problems at all follow-up points were considered to have stable resolutions, either resolved abstinent (RA) or resolved nonabstinent (RNA). Those who engaged in any problem drinking were considered to have unstable resolutions (UR), even if they later abstained or moderated.

Table 1 summarizes the sample characteristics as a function of participants' postresolution drinking status based on all follow-up data. During the initial resolution period required for inclusion, 89.1% of participants had abstained continuously and 10.9% had engaged in moderate drinking. Most participants' current drinking goal choice was informed by personal experience; 85% had made one or more serious resolution attempts in the past, with moderation attempts outnumbering abstinence attempts by about 4:1. During the present 1- to 2-year follow-up, 47.0% of participants maintained continuous or nearly continuous abstinence, 14.9% engaged in some moderate drinking with no problem drinking, and 38.1% engaged in problem drinking at some point. Of those who drank moderately, women consumed a mean of 27.5 ml of 190-proof ethanol per drinking day (SD = 8.56), and men consumed a mean of 38.9 ml (SD = 19.0), which fall within NIAAA (2005) gender-adjusted guidelines for low-risk drinking. Of those who drank heavily, over half relapsed during the first postresolution year. Mean quantities consumed per postresolution drinking day were 93.8 ml (SD = 51.1) for women and 123.8 ml (SD = 75.0) for men. During the first postresolution year, the mean and median number of drinking days for RNA participants was 73.93 (SD = 111.51) and 5.50 days, respectively, and the mean and median for UR participants was 44.23 (SD = 67.23) and 20.0 days, respectively.

Procedure

A trained interviewer conducted 1.5- to 3.0-hour individual interviews at baseline and at the annual follow-up points. After giving written informed consent, participants were administered a noninvasive breath test (Alco-Sensor III; Intoximeters, Inc., St Louis, MO) to verify sobriety. All predictors were derived from the initial interview, which covered drinking practices, life contexts, and monetary allocation during the year before participants' recent resolution up to the time of the interview. The follow-up assessments covered the time since the last interview. Brief phone interviews conducted midway between the annual follow-ups assessed drinking and help-seeking status and maintained contact. Participants received $40 for each annual interview and completion of questionnaires that they returned by mail, $10 for each phone interview, and a $50 bonus if they completed all assessments. The procedures that provided the data for the analyses are summarized in the next section and in Tucker et al. (2002, 2006, 2008).

Drinking practices and money spent on alcohol

Established TLFB procedures (Sobell & Sobell, 1992) were used to assess daily drinking practices during the preresolution year and again at each annual follow-up point. Participant reports of ounces of beer, wine, and liquor intake were converted to milliliters of 190-proof ethanol for analysis. Participants also reported how much money they spent each day on alcoholic beverages, regardless of whether the beverages were consumed. This was not excessively difficult because alcoholic beverages are sold in standard quantities, and problem drinkers typically buy and consume large quantities of a limited range of preferred beverages. As needed, TLFB interviewing techniques were used to facilitate reports of money spent on alcohol (e.g., use of anchor events, identification of sustained behavior patterns).

Monetary allocation

Participants reported their monetary income and expenditures during the same periods using an expanded set of commodity classes derived from U.S. federal consumer expenditure surveys (Vuchinich & Tucker, 1996). They were instructed to bring in financial records (e.g., bank records, paycheck stubs), and documented information was recorded first; 59.4% of participants provided some financial records. Then TLFB interviewing techniques were used to complete the financial assessment. Income in dollars was reported by source (e.g., work income, unemployment benefits, pensions, loans). Expenditures were reported in three general categories, each with subcategories, including housing (e.g., mortgage, rent, utilities), consumable goods (e.g., food, tobacco, alcohol), and other (e.g., entertainment, transportation, loan payments, money saved). Reports in each category typically involved many transactions during the assessment interval, which were summed to obtain category totals for analysis. In addition to direct verification using financial records, internal consistency and reliability checks on participants' reports of monetary allocation patterns supported their accuracy (reported in Tucker et al., 2006).

As described in Tucker et al. (2002, 2006, 2008), expenditures during the preresolution year were separated into obligatory and discretionary categories. Obligatory expenditures were for essential, ongoing, and largely fixed costs of living, including housing, food, transportation, medical, loan, and automatic payroll deductions (e.g., taxes, retirement, health insurance). Discretionary expenditures were for less essential commodities that could be purchased intermittently, including recreation, entertainment, alcohol, tobacco, other consumable goods, gifts, and elective savings. The ASDE index was computed as the proportion of discretionary expenditures summed over the preresolution year spent on alcohol minus the proportion of preresolution discretionary expenditures put into savings. ASDE values could range from 1.0 to –1.0, with lower scores representing proportionally less spending on alcohol and more on savings.

Questionnaires

After each interview, participants completed questionnaires that assessed moderation predictors in addition to those assessed during screening. Self-efficacy expectations to resist urges to drink heavily in high-risk situations were assessed using the Situational Confidence Questionnaire (SCQ; Annis & Graham, 1988), and health status was assessed using the health portion of the Health and Daily Living Form (HDL; Moos, 1985). Questionnaires were scored using established methods. Table 1 presents the total scores from the initial assessment.

Checks on data quality

In every study, in addition to checks on participants' financial reports, their reports relevant to the inclusion criteria and follow-up drinking status were assessed through collateral interviews or participant reliability checks when collaterals were unavailable. These data, summarized here, were reported previously (Tucker et al., 2002, 2006; Tucker, Foushee, Black, & Roth, 2007). Summed across studies, collaterals were interviewed at least once for 75.61% of the enrolled sample. Participant data were excluded when collaterals failed to verify participant reports relevant to the inclusion criteria or their drinking status during the follow-up. This rarely occurred (less than 2% of the initial sample of 253). For participants' retained for analysis, good to excellent agreement levels were found for drinking dimensions that could be directly observed by collaterals (e.g., alcohol-related problems, types of beverages consumed, date of initial resolution). The reliability of participant reports of drinking practices and money spent on alcohol also was examined for participants in the IVR study and was found to be excellent (Tucker et al., 2007). These findings strongly suggest that participants retained in the sample were reporting accurately.

Statistical Analysis

The mutually exclusive outcome groups were based on participants' drinking practices and problems over the entire follow-up interval: RA (n = 95)—continuous abstinence; RNA (n = 30)—some low-risk drinking with no problem drinking; or UR (n = 77)—one or more drinking episodes that exceeded the moderation criteria at any point. These conservative operational definitions deliberately separated recovering problem drinkers who resumed alcohol consumption into outcome groups on the basis of whether they engaged in any high-risk drinking, regardless of their terminal outcome status.

To evaluate the main behavioral economic hypotheses, the preresolution year monetary allocations to alcohol and savings, computed as a proportion of discretionary expenditures, were first examined as a function of drinking outcome status in a 3 (outcome status) × 2 (allocation type) analysis of variance (ANOVA) with repeated measures on the second factor. The Outcome × Allocation interaction effect from this analysis was used to determine whether the difference in these allocations, which constituted the ASDE index, was significantly related to outcome group membership. Once confirmed, we used a series of three-group multinomial logistic regression analyses to examine the utility of the ASDE index in predicting outcome group membership in relation to established predictors, including measures of problem severity (MAST, ADS, HDL physical health subscale, problem duration, help-seeking history), TLFB reports of preresolution drinking (days well functioning [abstinent and light drinking days combined], mean milliliters of ethanol consumed per drinking day), self-efficacy to resist heavy drinking (SCQ), and demographic characteristics.

Because these analyses focused on identifying predictors of moderated outcomes, the RNA group was the referent group so that the results yielded RA versus RNA and UR versus RNA contrasts and associated odds ratios (ORs) that indicated effect sizes. Although the limited RNA participants (30) prohibited comprehensive multivariable models that included all predictors simultaneously (Peduzzi et al., 1996), the RNA sample was sufficient to maintain a favorable event-to-variable ratio in a series of multinomial logistic regressions that included, first, the ASDE index alone, and then, in subsequent models, the ASDE index plus one other predictor. The latter analyses determined if the predictive utility of the ASDE index was independent of the predictive effects of each established predictor. Significant effects from the two-predictor models were then used to construct three-predictor models that evaluated whether the ASDE index continued to predict RNA outcomes beyond significant problem severity and drinking quantity measures. A final four-variable model examined whether the ASDE continued to have unique predictive utility when three other significant predictors from the three-variable models were included simultaneously. Continuous predictor variables in all logistic regression models were standardized to have a mean of 0 and a standard deviation of 1. The ORs and associated 95% confidence intervals (CIs) were based on a 1-standard deviation change in the predictors and allowed direct comparisons across predictors.

Tests of the Behavioral Economic Hypotheses

Table 1 summarizes univariate differences between the three outcome groups for established moderation predictors, and Table 2 summarizes group differences for the expenditure data from the preresolution year, including the ASDE index and expenditure components from which it was derived. The 3 × 2 ANOVA on the proportions of discretionary expenditures indicated an expected allocation main effect, F(1, 199) = 78.56, p < .0001, which reflected greater overall proportional allocation to drinking than savings prior to resolution. More important, as shown in Figure 1, a significant interaction effect was obtained that supported the hypotheses, F(2, 199) = 11.12, p = .0001. Comparisons using Tukey's honestly significant difference test showed that, as predicted, RNA participants had significantly lower discrepancies between alcohol and savings allocation proportions than both UR participants (p < .05) and RA participants (p < .05). RNA participants allocated proportionally less discretionary spending to alcohol and more to savings compared with UR participants (ps < .05) and less to alcohol than RA participants (p < .05).

Behavioral Economic Variables Based on Preresolution Monetary Allocation Patterns as a Function of Resolution Status at Follow-Up

Figure 1. Resolution Status × Allocation Type interaction among components of the Alcohol-Savings Discretionary Expenditure (ASDE) index based on the proportion of discretionary expenditures for alcohol and savings during the year prior to resolution onset (y axis). The error bars represent the standard errors of the drinking outcome group means.

The multinomial logistic regression analysis that included the ASDE index as the sole predictor revealed significant effects for both the UR versus RNA contrast (OR = 2.89, 95% CI = 1.77, 4.73, p < .0001) and the RA versus RNA contrast (OR = 1.93, 95% CI = 1.23, 3.02, p = .004). The OR from the first contrast indicated that a 1-standard-deviation increase in the ASDE index was associated with a 2.89-fold increase in the odds of resuming problem drinking compared with stable moderation. The OR from the second contrast indicated that a 1-standard-deviation increase in the ASDE index was associated with a 1.93-fold increase in the odds of stable abstinence compared with moderation.

The preceding logistic regression used drinking outcome assignments based on all available data from the 202 participants who had an ASDE score and at least 1 year of follow-up data. When this analysis was restricted to participants who were followed for 2 years (n = 152) and thus provided the longest continuous behavioral records, the ASDE index remained a significant predictor of both the UR versus RNA contrast (OR = 2.79, CI = 1.56, 5.00, p = .0006) and the RA versus RNA contrast (OR = 1.95, CI = 1.14, 3.32, p = .014). The same pattern of results was observed in an additional sensitivity analysis (n = 193) that excluded 4 RNA and 5 UR participants who were mostly abstinent but occasionally drank either moderately or heavily (UR vs. RNA: OR = 2.55, CI = 1.54, 4.21, p = .0003; RA vs. RNA: OR = 1.74, CI = 1.10, 2.75, p = .017). Consistent with the main behavioral economic hypothesis, frequent moderate drinkers had the lowest and frequent heavy drinkers had the highest mean ASDE scores.

Overall, these results supported the hypotheses concerning the ASDE index. The composite index separated the RNA group from the RA and UR groups, which were more similar.

Predictive Utility of the ASDE Index Relative to Established Moderation Predictors

Table 3 summarizes the results of logistic regressions that included the ASDE index and one other moderation predictor and also presents the correlations between the ASDE index and the other predictors. Four findings are noteworthy. First, the ASDE index showed low to modest correlations with all other predictors, ranging from .00 to .36 (rs > .14 or < −.14 were significant at p < .05), indicating that the ASDE was largely unrelated to the other predictors and capable of contributing new information to the prediction of outcomes in the multivariable models. Second, the analyses replicated many established moderation predictors, including lower dependence, fewer psychosocial and health problems, shorter problem durations, lower quantities consumed on drinking days, absence of help seeking, stable sociodemographic characteristics, and higher self-efficacy. Third, when the two-predictor models were run, the ASDE index remained a robust predictor of outcomes among the subset of participants who drank during the follow-up. The UR versus RNA contrast was significant in all models that included another predictor, indicating that the ASDE index explained unique variance after accounting for the other predictor. Fourth, the ASDE index distinguished outcomes among the subset of participants who maintained resolution. The RA versus RNA contrast was significant for the index when it was included with another predictor in all but two models that included either the MAST or mean milliliters per drinking day.

Multinomial Logistic Regressions Using the ASDE Index and One Established Moderation Predictor

On the basis of these results, additional three-variable logistic regressions were conducted that included the ASDE index and two other variables of conceptual interest or empirical utility. The MAST and ADS were not included in the same model because they were highly correlated (r = .68, p < .001) and would, therefore, be largely redundant in the same model. Table 4 presents the results of the three-variable models that we examined. In all eight complex models, the ASDE index continued to separate the UR and RNA groups. The index was not highly effective in separating the RA and RNA groups, although trends that approached significance for the ASDE were observed in Models 1, 3, and 8. A positive help-seeking experience and mean milliliters ethanol/drinking day consistently separated all three outcome groups, with absence of help seeking and lower quantities consumed being associated with an RNA status. The MAST, ADS, and SCQ contributed significantly to the separation of the UR and RNA groups. The MAST and ADS also separated the RA and RNA groups in models that did not include drinks per drinking day, but inclusion of that variable attenuated their predictive utility for the RA–RNA contrast.

Multinomial Logistic Regression Models Using the ASDE Index With Two Moderation Predictors

A final, comprehensive four-variable model was constructed from the strongest predictors identified in Table 3, namely the MAST, SCQ, mean milliliters ethanol/drinking day, and ASDE index. For the UR–RNA contrast, the SCQ (OR = 0.30, CI = 0.12, 0.74, p = .009), mean milliliters ethanol/drinking day (OR = 7.26, CI = 1.77, 29.74, p = .006), and ASDE (OR = 2.46, CI = 1.27, 4.77, p = .008) were statistically significant unique predictors, whereas for the RA–RNA contrast, the MAST (OR = 1.96, CI = 1.03, 3.72, p = .04) and mean milliliters ethanol/drinking day (OR = 9.54, CI = 2.37, 38.33, p = .002) were the statistically significant unique predictors.

The findings add to evidence supporting a behavioral economic analysis of drinking resolutions and extend the utility of a measure of preference for alcohol derived from preresolution spending patterns to predict moderation. Stable resolutions involving moderate alcohol use over 1- to 2-year follow-ups were associated with proportionally more preresolution discretionary monetary allocation to savings and less to alcohol compared with continuously abstinent resolutions and unstable resolutions that involved some problem drinking. Lower ASDE values presumably reflect more balanced monetary allocations between short-term and longer term objectives, suggesting that the temporal intervals over which problem drinkers organize and allocate their behavior, even while drinking heavily, may help identify those most able to make a transition to stable moderate use.

The support for the ASDE index in this re-analysis of pooled data from prior studies was obtained in conjunction with results that replicated established moderation predictors. As found during the controlled drinking debate (cf. Marlatt, 1983), moderation was associated with greater social stability and with lower problem severity, including shorter drinking histories, lower alcohol dependence and quantities consumed per drinking day, and fewer alcohol-related psychosocial problems (Miller & Munoz, 2005; Rosenberg, 2004). Higher self-efficacy to resist heavy drinking in high-risk situations also predicted moderation, which replicated recent findings that added this variable to those identified during the controlled drinking debate (Saladin & Santa Ana, 2004).

When included in models with other significant predictors, the ASDE index added unique information to the prediction of moderation and was especially effective in distinguishing outcomes among participants who engaged in postresolution drinking. The ASDE index was significant for the RNA–UR contrast in all models evaluated. Drinkers who maintained moderation had lower ASDE, MAST, and ADS scores and higher self-efficacy scores than those who relapsed.

The ASDE index also predicted outcomes among participants who remained resolved, separating the abstinent and moderation groups in models that included the ASDE alone or with one other predictor. The index was less effective in separating these groups in complex models that included quantities consumed, an attenuation that may be due in part to heterogeneity in the RA group. Some abstainers may be able to drink moderately but have not exposed themselves to postresolution alcohol use, whereas others might resume problem drinking. This unobserved moderation versus relapsed outcome among abstainers can, therefore, limit the full predictive significance of individual predictors of moderation outcomes. Given this attenuated separation, for purposes of choosing an initial abstinence or moderation drinking goal, it seems prudent clinically to require multiple favorable indicators of likely success at moderation until further research can establish decision-making algorithms that satisfactorily separate all three outcome groups. The present findings suggest that supplementing the MAST, ADS, and SCQ with questions about consumption quantities on drinking days and money spent on alcohol and put into savings provides a sound basis for making clinical judgments about initial drinking goal choice.

The ASDE findings also have implications for behavioral economic research and addictive behavior change applications. Evidence is accumulating that addictive behaviors are characterized by a foreshortened view of the future and that successful behavior change will likely involve a shift from a shorter to a longer view of the future and organizing behavior accordingly. In addition to the present support based on money allocation patterns in the natural environment, behavioral economic research on temporal discounting of hypothetical money and health outcomes has consistently found steeper discount functions in smokers, problem drinkers, opiate addicts, and gamblers (Bickel & Marsch, 2001). Social psychological studies have similarly found that the time perspectives of substance abusers are more present oriented and less future oriented than for normal controls (Henson, Carey, Carey, & Maisto, 2006; Keough, Zimbardo, & Boyd, 1999). Addiction treatment outcomes also are predicted by behavioral impulsivity measures that span laboratory and naturalistic assessments, including delay discounting of hypothetical rewards (e.g., Yoon et al., 2007), demand curve analysis based on hypothetical alcohol purchase tasks (MacKillop & Murphy, 2007), questionnaire measures of relative reinforcement value (e.g., Murphy, Correia, Colby, & Vuchinich, 2005; Schmitz, Sayre, Hokanson, & Spiga, 2003), and experiential discounting tasks that assess delay discounting using real, rather than hypothetical, monetary rewards (Krishnan-Sarin et al., 2007).

Such measures of behavioral impulsivity developed within a behavioral economic framework guided by the matching law (Herrnstein, 1970) show inconsistent relationships with personality questionnaires that assess traitlike impulsive tendencies, and the latter measures have no utility in predicting treatment outcomes (e.g., Krishnan-Sarin et al., 2007). Although it remains to be determined the extent to which the behavioral economic measures assess common or different dimensions of behavioral impulsivity and the relative reinforcing efficacy of substances (MacKillop & Murphy, 2007; Reynolds, Richards, Horn, & Karraker, 2004), they appear to measure functional changes in preferences for substance use and nondrug alternatives that are at the core of the dynamic addictive process. The ASDE index as currently assessed is decidedly among the more molar behavioral economic measures and appears to measure the relative reinforcement value of alcohol in the context of resource allocation to commodities available over different temporal intervals. Although the ASDE may prove to be less sensitive to short-term preference shifts than are brief measures of the current demand for drugs, it provides a comprehensive benchmark based on behavior patterns in the natural environment against which to evaluate the utility of briefer measures of relative preferences, including laboratory preparations.

Regardless of whether foreshortened views of the future are a cause or consequence of the addictive process, or both, it seems likely that interventions may facilitate positive change by promoting contact with the set of delayed positive consequences, or the sober “consumption bundle,” that typically flows from a sober lifestyle. Behavior patterns with delayed positive consequences often are more valuable as a whole compared with discrete acts chosen day-to-day (Rachlin, 1995), and the likelihood of maintaining longer term, higher yield patterns is presumably greater once contact is made with the delayed positive consequences. Drawing attention to delayed consequences or signaling their future availability (e.g., via self-monitoring, motivational interviewing, or decisional balance exercises) is one way to shift behavioral organization toward the future. Such approaches may reduce the appeal of short-term discrete rewards, like substance use, by helping people frame choices as involving an extended series of linked behaviors, events, and outcomes with higher overall value (Chapman, 1996; Rachlin, 1995).

The present research has limitations that merit attention in future studies. First, despite pooling across studies, the number of participants with stable moderation resolutions was still relatively small, which limited the number of predictors that could be evaluated simultaneously in multivariate models. Although quite coherent across models, the results merit replication with a larger sample of moderate drinkers. Second, because two of the earlier studies (Tucker et al., 2002, 2006) included too few moderate drinkers to analyze them separately from abstainers, the positive ASDE findings in those studies may have been amplified by including RNA participants in the stable resolution group along with RA participants. However, RNA participants constituted less than 12% of the stable resolution groups in these studies, suggesting that the overall ASDE results and interpretation were appropriate. Third, future studies of resolution should expand the window of selection to include problem drinkers who have resolved quite recently or are contemplating doing so. Participants with drinking problems that fall short of clinical diagnostic criteria also should be studied, and drinking-related inclusion criteria should be relaxed to allow research on outcomes that fall short of stable moderation but entail substantial reductions in drinking and related harm.

Pursuing the public health implications of understanding pathways to and predictors of moderation will require expanding the scope of research to include new concepts and methods, such as those provided by behavioral economics. Better characterizing the moderate use of alcohol by individuals with a history of problem drinking may provide new insights about the behavior regulation processes involved in resolution and relapse and help guide innovative behavior change strategies that increase contact with the population with problems.

Ainslie, G. (1975). Specious reward: A behavioral theory of impulsiveness and self-control. Psychological Bulletin, 82, 463–496.

American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Annis, H. M., & Graham, J. M. (1988). Situational Confidence Questionnaire (SCQ-39) user's guide. Toronto, Ontario, Canada: Addiction Research Foundation.

Bickel, W. K., & Marsch, L. A. (2001). Toward a behavioral economic understanding of drug dependence: Delay discounting processes. Addiction, 96, 73–86.

Cahalan, D. (1970). Problem drinkers: A national survey. San Francisco: Jossey-Bass.

Chapman, G. B. (1996). Temporal discounting and utility for health and money. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 771–791.

Green, L., & Kagel, J. H. (Eds.). (1996). Advances in behavioral economics: Vol. 3. Substance use and abuse. Norwood, NJ: Ablex.

Henson, J. M., Carey, M. P., Carey, K. B., & Maisto, S. A. (2006). Associations among health behaviors and time perspective in young adults: Model testing with boot-strapping replication. Journal of Behavioral Medicine, 29, 127–137.

Herrnstein, R. J. (1970). On the law of effect. Journal of the Experimental Analysis of Behavior, 13, 243–266.

Keough, K. A., Zimbardo, P. G., & Boyd, J. N. (1999). Who's smoking, drinking, and using drugs? Time perspective as a predictor of substance use. Basic and Applied Social Psychology, 21, 149–164.

Krishnan-Sarin, S., Reynolds, B., Duhig, A. M., Smith, A., Liss, T., McFetridge, A., Cavallo, D. A., et al. (2007). Behavioral impulsivity predicts treatment outcome in a smoking cessation program for adolescent smokers. Drug and Alcohol Dependence, 88, 79–82.

MacKillop, J., & Murphy, J. G. (2007). A behavioral economic measure of demand for alcohol predicts brief intervention outcomes. Drug and Alcohol Dependence, 89, 227–233.

Marlatt, G. A. (1983). The controlled drinking controversy: A commentary. American Psychologist, 38, 1097–1110.

Marlatt, G. A. (1985). Lifestyle modification. In G. A.Marlatt & J. R.Gordon (Eds.), Relapse prevention: Maintenance strategies in the treatment of addictive behaviors (pp. 280–348). New York: Guilford.

Miller, W. R., & Munoz, R. F. (2005). Controlling your drinking: Tools to make moderation work for you. New York: Guilford.

Moos, R. H. (1985). Health and Daily Living Form. In D. J.Lettieri, J. E.Nelson, & M. A.Sayers (Eds.), Alcoholism treatment assessment research instruments (NIAAA Treatment Handbook Series 2, pp. 275–292). Washington, DC: U.S. Government Printing Office.

Murphy, J. G., Correia, C. J., Colby, S. M., & Vuchinich, R. E. (2005). Using behavioral theories of choice to predict drinking outcomes following a brief intervention. Experimental and Clinical Psychopharmacology, 13, 93–101.

National Institute on Alcohol Abuse and Alcoholism. (2005). Helping patients who drink too much: A clinician's guide. Rockville, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Institute on Alcohol Abuse and Alcoholism.

Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 48, 1373–1379.

Premack, D. (1965). Reinforcement theory. In D.Levine (Ed.), Nebraska Symposium on Motivation (Vol. 13). Lincoln: University of Nebraska Press.

Rachlin, H. (1971). On the tautology of the matching law. Journal of the Experimental Analysis of Behavior, 15, 249–251.

Rachlin, H. (1995). Self-control: Beyond commitment. Behavioral and Brain Sciences, 18, 109–159.

Rachlin, H., Battalio, R., Kagel, J., & Green, L. (1981). Maximization theory in behavioral psychology. Behavioral and Brain Sciences, 4, 371–417.

Reynolds, B., Richards, J. B., Horn, K., & Karraker, K. (2004). Delay discounting and probability discounting as related to cigarette smoking status in adults. Behavioural Processes, 65, 35–42.

Rosenberg, H. (2004). International research—Target groups. In H.Klingemann, R.Room, H.Rosenberg, S.Schatzmann, L.Sobell, & M.Sobell (Eds.), Kontrolliertes Trinken als Behandlungsziel—Bestandesaufnahme des aktuellen Wissens [Controlled drinking: A review] (Research report F03–90, pp. 69–79). Bern, Switzerland: University of Applied Sciences, School of Social Work.

Saladin, M. E., & Santa Ana, E. J. (2004). Controlled drinking: More than just a controversy. Current Opinions in Psychiatry, 17, 175–187.

Schmitz, J. M., Sayre, S. L., Hokanson, P. S., & Spiga, R. (2003). Assessment of the relative reinforcement value of smoking and drinking using a multiple-choice measurement strategy. Nicotine & Tobacco Research, 5, 729–734.

Selzer, M. L. (1971). The Michigan Alcoholism Screening Test: The quest for a new diagnostic instrument. American Journal of Psychiatry, 127, 1653–1658.

Skinner, H. A., & Horn, J. L. (1984). Alcohol Dependence Scale (ADS) user's guide. Toronto, Ontario, Canada: Addiction Research Foundation.

Sobell, L. C., & Sobell, M. B. (1992). Timeline followback: A technique for assessing self-reported alcohol consumption. In R.Litten & J.Allen (Eds.), Measuring alcohol consumption (pp. 41–72). Totowa, NJ: Humana Press.

Sobell, L. C., Sobell, M. B., & Toneatto, T. (1992). Recovery from alcohol problem without treatment. In N.Heather, W. R.Miller, & J.Greeley (Eds.), Self-control and addictive behaviors (pp. 198–242). New York: Macmillan.

Tucker, J. A. (2003). Natural resolution of alcohol-related problems. In M.Galanter & J. P.Allen (Eds.), Recent developments in alcoholism: Vol. 16. Research in alcoholism treatment (pp. 77–90). New York: Plenum Press.

Tucker, J. A., Foushee, H. R., & Black, B. C. (2008). Behavioral economics of natural resolution of drinking problems using IVR self-monitoring. Experimental and Clinical Psychopharmacology, 16, 332–340.

Tucker, J. A., Foushee, H. R., Black, B. C., & Roth, D. L. (2007). Agreement between prospective IVR self-monitoring and structured retrospective reports of drinking and contextual variables during natural resolution attempts. Journal of Studies on Alcohol and Drugs, 68, 538–542.

Tucker, J. A., Vuchinich, R. E., Black, B. C., & Rippens, P. D. (2006). Significance of a behavioral economic index of reward value in predicting problem drinking resolutions. Journal of Consulting and Clinical Psychology, 74, 317–326.

Tucker, J. A., Vuchinich, R. E., & Rippens, P. D. (2002). Predicting natural resolution of alcohol-related problems: A prospective behavioral economic analysis. Experimental and Clinical Psychopharmacology, 10, 248–257.

Vuchinich, R. E., & Tucker, J. A. (1996). The molar context of alcohol abuse. In L.Green & J. H.Kagel (Eds.), Advances in behavioral economics: Vol.3. Substance use and abuse (pp. 133–162). Norwood, NJ: Ablex.

Vuchinich, R. E., & Tucker, J. A. (1998). Choice, behavioral economics, and addictive behavior patterns. In W. R.Miller & N.Heather (Eds.), Treating addictive behaviors: Processes of change (2nd ed., pp. 93–104). New York: Plenum.

Vuchinich, R. E., Tucker, J. A., & Harllee, L. M. (1988). Behavioral assessment. In D. M.Donovan & G. A.Marlatt (Eds.), Assessment of addictive behaviors (pp. 203–223). New York: Wiley.

World Health Organization. (2000). International guide for monitoring alcohol consumption and related harm (WHO/MSD/MSB/00.4). Geneva: World Health Organization, Noncommunicable Diseases and Mental Health Cluster, Department of Mental Health and Substance Dependence.

Yoon, J. H., Higgins, S. T., Heil, S. H., Sugarbaker, R. J., Thomas, C. S., & Badger, G. J. (2007). Delay discounting predicts postpartum relapse to cigarette smoking among pregnant women. Experimental and Clinical Psychopharmacology, 15, 176–186.

Submitted: December 27, 2007 Revised: December 5, 2008 Accepted: December 16, 2008

A Communication-Based Intervention for Nonverbal Children With Autism: What Changes? Who Benefits?

Acknowledgement: We thank all the participating schools, children, and their parents; PECS Consultants Sue Baker and Teresa Webb from Pyramid UK; and The Three Guineas Trust for their generous support of this project.

Over the past two decades, there has been accumulating evidence for the effectiveness of psychosocial programs for young children with autism. These tend to incorporate a mix of behavioral, developmental, and educational approaches, and although methods may vary, their general goals are to enhance cognitive, communication, and social skills while minimizing rigid and repetitive and other problem behaviors (see Lord & McGee, 2001; Rogers & Vismara, 2008, for reviews). In part due to the design of studies, however, there have been few opportunities to examine the detail of exactly what changes and who benefits as a result of these interventions.

In terms of measuring what changes, many early autism intervention studies, in particular within the applied behavior analysis field, used global measures of outcome such as IQ scores and school placement (Dawson & Osterling, 1997; Howlin, Magiati, & Charman, 2009). More recent studies have used a wider range of measures, including standardized tests of adaptive behavior, expressive and receptive language, and measures of autism severity (Howard, Sparkman, Cohen, Green, & Stanislaw, 2005; Remington et al., 2007). Some studies have attempted to include measures of change in skills or behaviors specifically targeted by the intervention, for example, parents' or carers' knowledge about autism (Jocelyn, Casiro, Beattie, Bow, & Kneisz, 1998). Furthermore, a small number of studies include naturalistic or quasinaturalistic measures of communication, for example, observing spontaneous communication and language in parent–child interactions (Aldred, Green, & Adams, 2004); observing parents' use of facilitative strategies during social interaction with their child (McConachie, Randle, Hammal, & Le Couteur, 2005); recording the frequency and rate of children's turn-taking, joint attention, and requesting behaviors (Yoder & Stone, 2006a, 2006b); and observing structured play and joint attention acts within parent– and experimenter–child interactions (Kasari, Freeman, & Paparella, 2006; Kasari, Paparella, Freeman, & Jahromi, 2008). In addition to increasing the face validity of the research, demonstration of change in specific behaviors is likely to be helpful in elucidating exactly how an intervention is working (Kazdin & Nock, 2003).

To examine who benefits from intervention, attempts have been made to study the effects of preintervention child characteristics on outcome, most notably IQ and age. For IQ, this usually involves examining correlations between preintervention IQ and postintervention outcome (Eldevik, Eikeseth, Jahr, & Smith, 2006; Harris & Handleman, 2000; Remington et al., 2007) or the comparison of outcomes for high-IQ versus low-IQ subgroups (Aldred et al., 2004; Ben Itzchak, Lahat, Burgin, & Zachor, 2008; Ben-Itzchak & Zachor, 2007). Significant positive associations have been found between preintervention IQ and outcome. However, Yoder and Compton (2004) highlighted the flaws of testing for moderators by exploring correlations or comparison of subgroups' effect sizes. They emphasized the importance of using statistical methods that enable differentiation of predictors of growth or progress from predictors of intervention response. Where participants have been randomly assigned to intervention or control conditions, the appropriate method for identifying predictors of intervention response is to test for statistical interactions between child characteristics and group assignment in relation to the outcome variables. Although such statistical tests of moderator effects are well established in medical trial literature and used increasingly in the psychiatry field (Kraemer, Wilson, Fairburn, & Agras, 2002) to date, they have only been used in two studies of autism intervention (Kasari et al., 2006, 2008; Yoder & Stone, 2006a, 2006b).

Kasari et al. (2006) randomized 58 preschool children with autism (aged 3–4) to either joint attention training, symbolic play training, or a control condition, demonstrating that both active interventions were effective at enhancing social communication skills. In a later study, they presented further analysis revealing that growth in expressive language was positively predicted by a number of joint attention and symbolic play variables (Kasari et al., 2008). Yoder and Stone (2006a, 2006b) used multiple regression to demonstrate the moderating effects of baseline joint attention abilities. Their randomized trial (N = 36) compared Picture Exchange Communication Training (PECS; Bondy & Frost, 1998) with Responsive Education and Prelinguistic Milieu Training (RPMT; Yoder & Warren, 1999). Children who initiated joint attention relatively more frequently at baseline benefited more from RPMT in terms of their postintervention frequency of joint attention initiations, whereas children who initiated joint attention less frequently at baseline benefited more from PECS (Yoder & Stone, 2006a). In a later analysis, using mixed-level modeling, they found object exploration also moderated intervention response (Yoder & Stone, 2006b). Thus, children who displayed object exploration behaviors more frequently at baseline benefited most from PECS, showing greater increases in production of nonimitative words at outcome. The children who showed lower object exploration at baseline benefited more with respect to word production if they had received RPMT.

These studies notwithstanding, to date relatively few studies of psychosocial intervention for children with autism have effectively investigated what changes in response to treatment and who benefits. Common use of standardized IQ and language assessments to measure outcome has meant there has been relatively little naturalistic analysis of change in the form and function of children's communication as a result of intervention. Insufficiently sized samples and lack of randomization have impeded investigation into treatment moderators.

Enhancing spontaneity in everyday communication has been highlighted as one of the most important goals of intervention for children with autism (Lord & McGee, 2001). The aim of PECS is to teach spontaneous and functional communication to children with autism in a social context (Bondy & Frost, 1998). PECS is widely used in home and educational settings (Preston & Carter, 2009; Sulzer-Azaroff, Hoffman, Horton, Bondy, & Frost, 2009), and there is evidence from a number of single-case and case-series studies (Charlop-Christy, Carpenter, Le, LeBlanc, & Kellet, 2002; Ganz & Simpson, 2004; Kravits, Kamps, Kemmerer, & Potucek, 2002; Magiati & Howlin, 2003; Schwartz, Garfinkle, & Bauer, 1998), a school-based controlled study (Carr & Felce, 2007), and two randomized controlled trials (RCTs) (Howlin, Gordon, Pasco, Wade, & Charman, 2007; Yoder & Stone, 2006a, 2006b) that PECS can lead to improved communication skills in this group.

Howlin et al. (2007) conducted a pragmatic RCT of PECS. The study was designed to test the “real world” effectiveness of PECS. Initial training was delivered to teachers and classroom assistants by PECS consultants at workshops. PECS was subsequently applied by school staff in the classroom under regular supervision by PECS consultants. Naturalistic observations of rates of children's communication were used as the primary outcomes to measure intervention effects. Immediately after training had ended (approximately 5 months), the rate at which children spontaneously initiated communication (IC) had significantly increased. Overall rates of children's use of picture cards (P) to communicate (i.e., spontaneous or prompted) had also significantly increased. By the 9-month follow-up, however, these effects had disappeared. Overall rates of children's speech/vocalization (S) (including spontaneous and prompted) did not increase.

In the present article, using the sample from Howlin et al. (2007), we explore exactly which communication forms were used by children more spontaneously as a result of PECS training, which communicative functions increased and which children benefited most from the intervention. We aimed to build on previous studies by applying appropriate analysis to this relatively large sample to address the following questions:

1. Did PECS training act specifically to increase children's spontaneous communication using the picture cards or did its effect generalize to support greater spontaneity using speech as well? Previous studies have demonstrated that PECS supports children to communicate spontaneously using picture cards (e.g., Bondy & Frost, 1994; Charlop-Christy et al., 2002; Ganz & Simpson, 2004; Kravits et al., 2002) and sometimes speech (e.g., Bondy & Frost, 1994; Charlop-Christy et al., 2002; Ganz & Simpson, 2004). Two studies using more naturalistic measures of outcome also suggest that PECS can increase spontaneous communication using picture cards and speech together (Carr & Felce, 2007; Yoder & Stone, 2006a).

2. Did PECS increase children's spontaneous communication purely for instrumental requesting or did the training also lead to increased spontaneous communication for more social purposes? PECS training initially focuses on teaching children to make requests for objects. Later training phases aim to broaden the range of communicative functions, such as sharing attention through commenting (Bondy, Tincani, & Frost, 2004). Studies have shown that PECS training can be used successfully to teach children spontaneous requesting for objects (Carr & Felce, 2007; Ganz & Simpson, 2004; Kravits et al., 2002), and some have demonstrated effects on other forms of noninstrumental, more social communication (Schwartz et al., 1998; Yoder & Stone, 2006b).

3. Which children benefited most from PECS training? PECS was specifically developed for children with autism to obviate the need for prerequisite communication skills. It might be hypothesized therefore that response to PECS training would not be predicted by such factors as language comprehension skills or autistic symptomatology. Within the autism intervention literature more generally, however, higher IQ has been associated with better outcome (e.g., Harris & Handleman, 2000; Schwartz et al., 1998). Yoder and Stone's (2006a, 2006b) studies have been the only systematic investigation of PECS response predictors to date, finding that children who were most impaired in terms of baseline language and joint attention skills were those who gained most from PECS training. Given the lack of research in this area, we took an exploratory approach in the present study, investigating the potential moderating effect on PECS training of four factors measured at baseline: chronological age, expressive language, autistic symptomatology, and cognitive ability.

Method

Participants of the RCT

Eighty-four children (73 boys, 11 girls) from 17 classes in special needs elementary schools participated in the study. Classes were broadly similar, with a child–adult ratio of approximately 2:1. Class teaching programs varied, but most classes adopted an eclectic approach incorporating a range of visual systems and structured teaching, often based on the TEACCH methodology (Mesibov, Shea, & Schopler, 2004). Picture cards were present in most classrooms in the treatment and nontreatment groups even at baseline, though these were not necessarily used according to PECS principles. All schools were situated in Greater London or the south east of England. Children were aged between 4 and 10 years (mean age at baseline = 6.8 years, SD = 1.26), and all had an intellectual disability. Ethnicity, socioeconomic status, and comorbidity data were not formally collected. This was a community-based study that included all suitable children whose parents consented. To be eligible, children had to (a) have a formal clinical diagnosis of autism, (b) use little or no functional language (i.e., no more than single words), (c) have no sensory impairment, and (d) be aged between 4 and 11 years and (e) not using PECS beyond Phase 1.

Informed consent and ethical approval

Written informed consent to participate in the study was obtained from the parent or guardian of each child and from a senior member of staff from each school. The original trial protocol was prospectively reviewed and approved by the Wandsworth Local Research Ethics Committee (Ref. IAS/der/02.42.6).

Describing the group at baseline

To obtain baseline data on autism severity, all children were assessed using Module 1 of the Autism Diagnosis Observation Schedule (ADOS; Lord et al., 2000). The ADOS is an interactive semistructured assessment of communication, social interaction, imagination, and repetitive and stereotyped interests. Assessment consists of a range of activities and social presses providing a standardized context in which to observe specific behaviors. There are four modules. Module 1 was used in this study, designed for use with preverbal individuals or for those whose expressive language is still at single word or simple phrase level. Seventy-five children met the ADOS criteria for a diagnosis of autism; nine children met criteria for autism spectrum disorder. Score from Item A1 of the ADOS was used as an index of expressive language ability (0 = regular use of utterance of two or more words; 1 = occasional phrases only, mostly single words; 2 = recognizable single words only; 3 = at least one word or word approximation but fewer than five words; 4 = no words or word approximations). Thirty-eight children (45%) used no words or word approximations during the ADOS, 31 (37%) used single words, and 15 (18%) used at least one phrase. Most children (64%) scored 0 on the Expressive One Word Picture Vocabulary Test (Brownell, 2000). Nonverbal developmental quotient (NVDQ = nonverbal mental age equivalent/chronological age ×100) was ascertained using the Mullen Scales of Early Learning (Mullen, 1995). Group median NVDQ was 29.90 (interquartile range was 21.20–40.52). In summary, the sample comprised children with clear autism and who were very impaired with regard to verbal and nonverbal skills.

Design of the RCT

As a group-randomized control trial, class groups (each including approximately six children and two to three staff) were assigned into one of three intervention groups. The immediate treatment group (ITG; five class groups, 26 children) received training immediately after the baseline assessment; the delayed treatment group (DTG; six class groups, 30 children) received training about 9 months later, immediately after Time 2 assessment; the no-treatment group (NTG; six class groups, 28 children) received no training. Staggering of treatment across two time periods maximized the number of children involved in the study and allowed investigation of the continued effectiveness of any immediate treatment effects noted. The data analyses incorporated each child contributing all measurements within all control, treatment, and posttreatment periods; thus, statistical power was not compromised by the three-arm approach to data collection. Differences between the three groups at baseline were analyzed and reported in Howlin et al. (2007). The analysis was designed to adjust for these differences. Figure 1 shows recruitment, the points at which intervention was delivered, and when each of the three groups was observed.

Figure 1. Flow chart illustrating sample selection, recruitment, training, and outcome assessment. Adapted from “The Effectiveness of Picture Exchange Communication System (PECS) Training for Teachers of Children With Autism: A Pragmatic, Group Randomised Controlled Trial,” by P. Howlin, R. Kate Gordon, G. Pasco, A. Wade, and T Charman, 2007, Journal of Child Psychology and Psychiatry, 48, pp. 473–481. Copyright 2007 by John Wiley and Sons.

Outcome measurement

The outcome measure was a 15-min videotaped observation, intended to be an ecologically valid measure of communication skills. Children were filmed in their class snack sessions at Time 1 (baseline) and twice further over a period of 20 months (2 academic years). Snack sessions were selected as these were likely to create the most opportunities for children to make spontaneous requests. Furthermore, daily snack sessions occurred in all the classes and were broadly similar. These sessions usually lasted approximately 15 min and involved all children and class staff sitting at tables in the classroom or school kitchen. Drinks and food snacks such as fruit or cookies were given out or were on offer for children to request. Where classes used picture cards, these were usually made available for children (e.g., by placing a large board at the front of the classroom or by handing out books with the cards inside).

Children's communication was coded from the videotape using an observation schedule designed specifically for this study (Classroom Observation Schedule for Measuring Intentional Communication; Pasco, Gordon, Howlin, & Charman, 2008). The primary outcome variable was frequency of child-initiated communication (IC). Frequencies of different communication modalities used (such as the number of times a child used a picture card [P] and/or speech/vocalization [S] to communicate) were also recorded; communication functions were recorded by counting each time a child communicated for the purpose of requesting objects (R) and for the purpose of requesting a social interaction or commenting (D). In this way, a single communication act might produce three or more codes, for example, as a spontaneous initiation (IC), of the use of a picture card (P), and for the purpose of requesting (R).

Data analysis

Where outcomes are numerical counts of relatively rare events, Poisson regression is a useful method for analysis (Dobson, 2002). The Poisson regression model expresses the log outcome rate as a linear function of a set of predictors. In this study, Poisson regression models were produced for each of the five outcome variables of interest, concerned with form or function of children's spontaneously initiated communication: spontaneous communicative initiation using picture cards (IC-P); spontaneous communication using speech (IC-S); spontaneous communication using both simultaneously (IC-PS); spontaneous communication to request for objects (IC-R), and spontaneous communication to request for social routine or commenting (IC-D). The regression models were created within the Stata IC Version 10 (StataCorp., 2003).

As can be seen in Figure 1, the data set comprised data from three time points in the three different experimental groups. Multilevel models were used that took account of the longitudinal nature of the measurements, time trends, differing treatment regimes within the same individuals over time, and within child correlations between repeated measurements (Goldstein, 2003). The standard errors of the model parameters were thus adjusted for any within-child (across time) or within-class (between children) correlations. Models also allowed adjustment for any group differences at baseline in terms of age, developmental level, expressive language, and autistic symptom severity. Each model included an independent binary intervention variable (i.e., intervention or no intervention), a further binary variable to denote follow-up (this occurred at Time 3 for the ITG group only), a time variable (continuous in order to adjust for differences in the actual lengths of time between observations, i.e., Time 1 = 0 days), and an offset to adjust for the difference in the lengths of snack times for individual children.

In Poisson regression, effect size is represented by the rate ratio (RR) that estimates the relative rate of change in the mean number of events attributable to each explanatory variable. For example, for a binary intervention variable, the RR represents the relative difference in mean frequency of spontaneous initiations for children in the intervention group compared with those not in the intervention group. For continuous variables (e.g., baseline age [months]), the RR represents the relative difference in the mean frequency of initiations for every increase in one unit of the explanatory variable—in this case, for every month older the child was at baseline. An RR of 1 indicates no change, and so, for example, an RR of 1.2 represents an increase of 20% for each unit increase; an RR of 0.7 represents a decrease of 30% for each unit increase. RRs for estimates from the five models (for each of the five outcome variables) are reported along with 95% confidence intervals and p values. These models were not independent and were interpreted jointly taking into account the relationship between the various outcomes.

Testing for intervention moderators

Where PECS had a significant effect, a second round of analyses was conducted in order to identify potential intervention response moderators. If baseline factors (i.e., chronological age, autistic symptomatology, expressive language, or developmental quotient) independently predicted progress at postintervention (shown in Table 2), tests for Intervention × Baseline Factor interactions were conducted to explore whether they also predicted a specific intervention response. The RR for the interaction term represents the impact of the baseline variable on the outcome over and above any existing variance due to a main intervention effect or a main effect of the baseline factor.

Rate Ratio Estimates (and 95% Confidence Intervals) for Each of the Five Outcome Variables

Results

The results are presented in two parts. First, we examined the impact of PECS training on children's spontaneous communication using three different communication modalities and for two different functions. Second, controlling for differences, we tested baseline variables for their potential moderating effect on the intervention.

Change in spontaneous communication following PECS training

Table 1 shows the median rate of initiations per minute for the five variables in each of the three treatment arms at each of the three time points. The rates in bold are immediately following PECS training in the ITG and DTG. The italicized rates are at 9-month follow-up (ITG only). These figures indicate some changes following PECS training. For example, in the DTG, the median rate of spontaneous initiation of communication using picture cards went from 0 to 0.44 per minute, that is, more than 6 times per 15-min snack session. In the ITG, the median rate of spontaneous communication using speech or vocalization went from 0.03 to 0.13 per minute, and in the DTG, the median rate of spontaneous requesting rose from 0.03 to 0.46 times per minute. Despite these group effects, for each of the form and function variables, some children remained at zero, even after PECS training. For example, of the 56 children in the ITG and DTG, 12 were still not using picture cards to spontaneously communicate at all after the training, and nine were still not making spontaneous requests.

Median Initiations Made Per Minute at Each Time Point for Each of the Three Treatment Groups

Table 2 shows the results of the Poisson analysis for each variable. RRs are shown for change attributable to the intervention immediately post-PECS training, at 9-month follow-up (for the ITG group only), and for each of the nonintervention variables measured at baseline. Initiations using picture cards (IC-P), using speech (IC-S), and using both simultaneously (IC-PS) all increased significantly following training (RR = 1.90, 95% CI [1.46, 2.48], p < .001; RR = 1.77, 95% CI [1.35, 2.32], p < .001; RR = 3.74, 95% CI [2.19, 6.37], p < .001, respectively). The average increase observed was similar in size for IC-P and IC-S and about twice as large for IC-PS. However, it should be noted that the confidence intervals are wide, and in all instances the data are compatible with a twofold increase in the RR, so we cannot necessarily infer that the effect is any greater for IC-PS. Spontaneous requesting for objects (IC-R) significantly increased following training (RR = 2.17, 95% CI [1.75, 2.68], p < .001), but requesting for social routine or commenting (IC-D) did not (RR = 1.34, 95% CI [0.83, 2.18], p = .237). Children in the ITG (n = 26) were observed again at follow-up (approximately 9 months after the end of the training period). Although the effect on spontaneous initiation using speech/vocalization (IC-S) had persisted (RR = 1.70, 95% CI [1.12, 2.58], p = .012), none of the other effects were significant (IC-P, RR = 0.69, 95% CI [0.41, 1.15], p = .15; IC-PS, RR = 1.90, 95% CI [0.76, 4.76], p = .17; IC-R, RR = 1.11, 95% CI [0.76, 1.62], p = .60).

Variables moderating the effect of PECS training

We analyzed baseline variables for their power to predict progress in general and to predict specific response to treatment. Seven baseline variables (shown in bold in Table 2) were independently and significantly related to general progress at postintervention and so were testable as potential moderators of the intervention effects. Of these seven Intervention × Baseline Variable interactions tested, two were found to be significant. The impact of the intervention on children's spontaneous initiation of communication using speech/vocalization (IC-S) was moderated by baseline autistic symptomatology (RR = 0.90, 95% CI [0.83, 0.98], p = .011). As can be seen in Figure 2, children whose autistic symptomatology score was lowest at baseline (i.e., least severe symptoms) showed the largest increases in spontaneous use of speech/vocalization following intervention. Each unit increase in ADOS score was associated with a 10% decrease in average rate of initiation using speech/vocalization (IC-S). Baseline expressive language did not moderate intervention effects for this outcome (RR = 1.05, 95% CI [0.90, 1.24], p = .524).

Figure 2. Graph showing the moderating effect of autistic symptomatology on the effect of PECS on children's spontaneous communication using speech/vocalization (Autism Diagnosis Observation Schedule [ADOS] scores are on a severity scale; higher score means more severe symptomatology).

Baseline expressive language moderated the effect of PECS training on children's spontaneous initiation using picture cards and speech/vocalization together (RR = 0.62, 95% CI [0.44, 0.88], p = .008). Expressive language was rated on a severity scale. Figure 3 shows that those children with the most expressive language at baseline (lower score represents better expressive language) showed the biggest increase in their use of picture cards and speech/vocalization together to spontaneously initiate communication. Each unit increase in expressive language deficit score was associated with a 38% decrease in the average rate of initiations using picture cards and speech together (IC-PS). Neither baseline developmental quotient nor autistic symptomatology moderated the effects of the intervention for this outcome (RR = 1.01, 95% CI [0.98, 1.05], p = .539 and RR = 0.94, 95% CI [0.79, 1.12], p = .477, respectively). As can be seen in Table 2, baseline developmental quotient (DQ) predicted of rate of initiation using picture card (IC-P) and rate of initiation for the purpose of instrumental requesting (IC-R) immediately postintervention, but interaction tests demonstrated that this did not moderate the effects of the training on these behaviors (for IC-P, DQ × Intervention, RR = 0.99, 95% CI [0.98, 1.01], p = .214; for IC-R, DQ × Intervention, RR = 0.99, 95% CI [0.98, 1.00], p = .212).

Figure 3. Graph showing the moderating effect of baseline expressive language on children's spontaneous communication using picture cards and speech/vocalization. ADOS = Autism Diagnosis Observation Schedule.

Discussion

PECS is recognized as an effective intervention for increasing communication in children with autism (Preston & Carter, 2009; Sulzer-Azaroff et al., 2009), and our RCT demonstrated specifically that PECS training can significantly enhance the spontaneity of children's communication (Howlin et al., 2007). In this article, we asked exactly how PECS training increased this communicative spontaneity and for which children. That is, we wanted to examine, first, whether the increased spontaneity was confined to communication using the picture symbols or whether PECS also impacted on the spontaneity of children's use of speech/vocalization. Second, we wished to examine whether the increased spontaneous communication was being used only for instrumental purposes (e.g., getting a snack) or whether children were also spontaneously initiating communication for more social purposes as a result of PECS training. Third, we wanted to identify factors that might be moderating the effect of PECS training and therefore predictive of which children might benefit most from the training. We used Poisson regression analysis to examine the children's spontaneous communication using different communication modalities and for different functions and to test for interactions between the intervention and baseline child variables.

The naturalistic and relatively fine-grained outcome measurement meant that it was possible to analyze exactly how PECS was enhancing children's spontaneous communication in an everyday situation. A small number of previous intervention studies have examined the form of children's communication but have not focused purely on spontaneous unprompted communication. The present analyses revealed that although PECS training did lead to children spontaneously communicating more using the picture cards, it also led to increased spontaneity in children's use of speech and their use of picture cards and speech in combination. The training appears to have increased spontaneous requesting for objects or help but not spontaneous requesting for social routine or commenting.

In contrast to some other reports (Bondy & Frost, 1994; Charlop-Christy et al., 2002), in our primary analysis of PECS RCT, we did not observe an effect of the intervention on overall use of speech (Howlin et al., 2007). The present analysis revealed, however, that PECS did enhance the use of speech as a modality to spontaneously initiate communication, as well as enhancing spontaneity using picture cards. So, although it would appear that PECS training did not enhance speech development per se, for those children who were already using some speech or vocalization, PECS appears to have provided a structure for them to use this mode to communicate without prompting. It would seem that PECS fostered spontaneity more generally across modalities rather than just acting to increase children's use of picture cards. Furthermore, the effect of PECS training on children's spontaneous speech/vocalization appears to have been particularly robust as it was also observed 9 months after the end of the training period in the group who received PECS training early on. There was no long-term effect on spontaneous use of picture cards.

Detailed analysis of the functions of spontaneous communication in autism intervention is also rare. In this study, analysis revealed a clear effect on children's spontaneous communication for the purposes of requesting for objects, such as a drink or a toy, which is the first communicative function taught through PECS teaching phases (Frost & Bondy, 2002). This replicates findings from earlier research (Schwartz et al., 1998; Yoder & Stone, 2006a, 2006b). There was no effect of training on children's spontaneous communication for social purposes. This might be due to the fact that the children in this sample had severe autism symptoms and, as a group, were very delayed with regard to verbal and nonverbal skills. Furthermore, the discrepancy between instrumental and social communication is perhaps to be expected given that the children were observed in class snack sessions. It is possible that observation of children in other nonsnack sessions might have revealed effects of training on communication for other noninstrumental purposes. Also, it is possible that if the training had persisted for longer or had been more intense, changes in spontaneous social, noninstrumental communication might have been seen. Some case study reports have described children successfully learning to communicate for social interaction purposes such as commenting (e.g., Schwartz et al., 1998; Webb, 2000), although, to date, no experimental trials have demonstrated this effect of PECS.

Two baseline variables appeared to moderate the effect of PECS training. First, less severe autistic symptomatology at baseline predicted the greatest increases in spontaneous speech. Second, higher level of expressive language at baseline predicted greater increases in spontaneous use of speech and picture cards together. This is to be expected, as more severe autism and greater language disability are not independent. Thus, the fact that the least severely autistic children and those with the most expressive language showed the greatest improvements in these areas is consistent with the autism intervention literature more generally (e.g., Harris & Handleman, 2000; Kasari et al., 2008). We observed no interactions between PECS training and any of the abilities measured at baseline on children's spontaneous use of the picture cards or spontaneous requesting. Yoder and Stone's (2006a,2006b) studies have been the only other systematic examination of moderators of PECS intervention. They compared PECS with RPMT, and although there was no overall difference between the interventions, children who were most impaired in baseline language and joint attention skills gained most in terms of their joint attention skills from PECS training, whereas the more able children made better progress with RPMT. The present study did not replicate the finding that the less able children benefited more from PECS. A potential explanation is that, as a group, the children in the present sample were less able than Yoder and Stone's sample. In Yoder and Stone's sample, the mean nonverbal mental age was 18.8 months (standard deviation 4.5 months) at 3 years of age (Table 1 in Yoder & Stone, 2006a, p. 429), meaning that the mean NVDQ was approximately 50, whereas the mean NVDQ in our sample was around 30.

Despite the fact that all children in the present study were very impaired in terms of their verbal and nonverbal skills, spontaneous use of pictures to communicate and spontaneous requesting did increase, and this was not predicted by better baseline language or less severe autism symptoms. This suggests that PECS training was equally accessible to these children in terms of teaching these skills specifically. This seems to support the idea that, beyond the need for some very basic cognitive skills required in order to exchange the cards (e.g., object permanence), few preexisting verbal or nonverbal skills are required to learn to use PECS (Bondy & Frost, 1998).

Strengths and limitations of the present study

The unique quality of the data presented here is that they are derived from an examination of “real world” effectiveness. The study took an inclusive approach to recruitment, aiming to include all suitable schools within a defined but large geographical area in the south east of England and included all suitable children whose parents consented. As the trial was community based, intervention was delivered to teachers and classroom staff via a workshop and follow-up visits to the schools. Teachers had to implement the program amidst all the other pressures and distractions of running a classroom for children with special educational needs and from a wide variety of backgrounds. Children were required to access the intervention in spite of their severe autistic symptoms, language impairments, and perhaps other comorbid problems. In other words, PECS training that was delivered and evaluated seemed to be a realistic representation of PECS training most children are likely to receive.

The design of the study and analysis ensured that the use of three treatment groups did not detract from the numbers effectively used for the intervention and no-intervention groups. The study was, in fact, strengthened by having within-individual comparisons (i.e., the delayed intervention group) as well as between-group comparisons over the same time frame. The use of the multilevel model allowed for efficient use of data in the three-treatment arm format adopted in this study, taking into account correlations between repeats from the same individuals and allowing for the serial nature of measurements under different treatment conditions. The design also enhanced the power to investigate the immediate effects of the intervention and enabled investigation of the longer term effects of the training. The incorporation of baseline data further strengthened the results.

The interaction analysis applied in this study is relatively novel to this field and demonstrates the possibility of using relatively sophisticated statistical models to test for moderator effects on interventions for children with autism. As has been discussed above, this has been rarely done in the autism field in the past, and thus there is very little reliable information about who benefits most from various interventions for children with autism.

As this trial was conducted primarily in schools, we had little direct contact with parents, aside from their consenting for their child to take part in the study. As a consequence, we did not collect systematic information on family variables such as ethnicity and other background factors (socioeconomic status, parental income) that might also be related to differential outcome, nor did we collect detailed information on potential school moderating factors. Instead, we focused on child characteristics as moderating factors. Generalization of these findings will require replication in samples with well-described demographic information as well as well-characterized schools/classrooms.

For logistical reasons (i.e., limited resources), we were limited to observing children in their classrooms. We opted to observe them during snack sessions as this was a session that created more opportunities for children to make spontaneous requests, relative to less structured sessions, and this was a common feature on the timetables of all classes involved in the study. However, the snack sessions are relatively brief periods, when children are usually highly motivated to make approaches for food, and so the data may not represent changes in children's communication in other less structured or less motivating contexts. Observations of children communicating in other class sessions or at home would have revealed the extent to which the observed effects generalized out of the relatively structured setting of class snack time.

With regard to the analysis of intervention response predictors, we were limited to testing four child factors measured at baseline. It is possible that other factors, not measured, were moderating the intervention effects, including those that were external to the children (i.e., environmental factors). It is likely that differences between the classes and the ways in which PECS was implemented also influenced children's progress. Treatment fidelity measures will be important for future studies of psychosocial interventions.

A limitation of the analysis of response predictors was that, although relatively sophisticated, essentially it was based on comparing subgroups, and the study was not primarily powered for this. The chance of Type II errors is thus increased. The results do, however, provide a good basis for further discussion. In the future, as more intervention studies are conducted and there is greater consistency in approach across research, pooling of samples may be possible, thus increasing statistical power for identifying invention response moderators.

Implications

In summary, the findings show that classroom-based PECS training enhances children's ability to make spontaneous instrumental requests not only using pictures but also using speech, or a combination of both. It also shows that, similar to other interventions, less impaired children appear to show the most improvement in these areas. Where these improvements were seen, they represented noticeable change in children's communication. For example, in one treatment group, the median rate of spontaneously initiated communication using PECS went from 0 times per 15-min snack session up to more than 6 times, and the median rate of spontaneous requesting rose from ~0.5 times to ~7 times per snack session. It is important to remember, however, that these figures are based on group effects. Such impressive gains were not seen in all children who received PECS training, and for some children, no gains were made at all. Nevertheless, for a child who has not been communicating at all to request even twice in a 15-min snack session represents a meaningful change.

The study also has important methodological implications. This article builds on the findings of one of the larger RCTs conducted in the autism field to date (Howlin et al., 2007; though see Aman et al., 2009; Green et al., 2010, for larger studies). The adequately sized sample, well described in terms of verbal and nonverbal abilities, provides the opportunity for analysis of the specific effects of psychosocial intervention for children with autism. The study demonstrates the feasibility of applying robust statistical techniques to pragmatic, “real world” trials in this field to elucidate exactly what changes and for whom.

Aldred, C., Green, J., & Adams, C. (2004). A new social communication intervention for children with autism: Pilot randomised controlled treatment study suggesting effectiveness. Journal of Child Psychology and Psychiatry, 45, 1420–1430. doi:10.1111/j.1469-7610.2004.00338.x

Aman, M. G., McDougle, C. J., Scahill, L., Handen, B., Arnold, L. E., Johnson, C., . . .Wagner, A. (2009). Medication and parent training in children with pervasive developmental disorders and serious behavior problems: Results from a randomized clinical trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 1143–1154. doi:10.1097/CHI.0b013e3181bfd669

Ben Itzchak, E., Lahat, E., Burgin, R., & Zachor, A. D. (2008). Cognitive, behavior and intervention outcome in young children with autism. Research in Developmental Disabilities, 29, 447–458. doi:10.1016/j.ridd.2007.08.003

Ben-Itzchak, E., & Zachor, D. A. (2007). The effects of intellectual functioning and autism severity on outcome of early behavioral intervention for children with autism. Research in Developmental Disabilities, 28, 287–303. doi:10.1016/j.ridd.2006.03.002

Bondy, A., & Frost, L. A. (1994). The Delaware Autistic Program. In S. L.Harris & J. S.Handleman (Eds.), Preschool education programs for children with autism (pp. 37–54). Austin, TX: Pro-Ed.

Bondy, A., & Frost, L. A. (1998). The Picture Exchange Communication System. Seminars in Speech and Language, 19, 373–389. doi:10.1055/s-2008-1064055

Bondy, A., Tincani, M., & Frost, L. (2004). Multiply controlled verbal operants: An analysis and extension to the picture exchange communication system. Behavior Analyst, 27, 247–261.

Brownell, R. (2000). Expressive One Word Picture Vocabulary Test. Novato, CA: Academic Therapy Publications.

Carr, D., & Felce, J. (2007). The effects of PECS teaching to phase III on the communicative interactions between children with autism and their teachers. Journal of Autism and Developmental Disorders, 37, 724–737. doi:10.1007/s10803-006-0203-1

Charlop-Christy, M. H., Carpenter, M., Le, L., LeBlanc, L. A., & Kellet, K. (2002). Using the picture exchange communication system (PECS) with children with autism: assessment of PECS acquisition, speech, social-communicative behavior, and problem behavior. Journal of Applied Behavior Analysis, 35, 213–231. doi:10.1901/jaba.2002.35-213

Dawson, G., & Osterling, J. (1997). Early intervention in autism: Effectiveness and common elements of current approaches. In M.Guralnik (Ed.), The effectiveness of early intervention: Second generation research (pp. 307–326). Baltimore, MD: Brookes Publishing.

Dobson, A. J. (2002). An introduction to generalized linear models (2nd ed.). London, England: Chapman & Hall.

Eldevik, S., Eikeseth, S., Jahr, E., & Smith, T. (2006). Effects of low-intensity behavioral treatment for children with autism and mental retardation. Journal of Autism and Developmental Disorders, 36, 211–224. doi:10.1007/s10803-005-0058-x

Frost, L., & Bondy, A. (2002). PECS: Picture Exchange Communication System training manual (2nd ed.). Cherry Hill, NJ: Pyramid Educational Consultants.

Ganz, J. B., & Simpson, R. L. (2004). Effects on communicative requesting and speech development of the Picture Exchange Communication System in children with characteristics of autism. Journal of Autism and Developmental Disorders, 34, 395–409. doi:10.1023/B:JADD.0000037416.59095.d7

Goldstein, H. (2003). Multilevel statistical models: Kendall's library of statistics3(3rd ed.). London, England: Hodder Arnold.

Green, J., Charman, T., McConachie, H., Aldred, C., Slonims, V., Howlin, P., . . .PACT Consortium. (2010). Parent-mediated communication-focused treatment in children with autism (PACT): a randomised controlled trial. The Lancet, 375, 2152–2160. doi:10.1016/S0140-6736(10)60587-9

Harris, S. L., & Handleman, J. S. (2000). Age and IQ at intake as predictors of placement for young children with autism: A four- to six-year follow-up. Journal of Autism and Developmental Disorders, 30, 137–142. doi:10.1023/A:1005459606120

Howard, J. S., Sparkman, C. R., Cohen, H. G., Green, G., & Stanislaw, H. (2005). A comparison of intensive behavior analytic and eclectic treatments for young children with autism. Research in Developmental Disabilities, 26, 359–383. doi:10.1016/j.ridd.2004.09.005

Howlin, P., Gordon, R. K., Pasco, G., Wade, A., & Charman, T. (2007). The effectiveness of Picture Exchange Communication System (PECS) training for teachers of children with autism: a pragmatic, group randomised controlled trial. Journal of Child Psychology and Psychiatry, 48, 473–481. doi:10.1111/j.1469-7610.2006.01707.x

Howlin, P., Magiati, I., & Charman, T. (2009). Systematic review of early intensive behavioral interventions for children with autism. American Journal on Intellectual and Developmental Disabilities, 114, 23–41.

Jocelyn, L. J., Casiro, O. G., Beattie, D., Bow, J., & Kneisz, J. (1998). Treatment of children with autism: A randomized controlled trial to evaluate a caregiver-based intervention program in community day-care centers. Journal of Developmental and Behavioral Pediatrics, 19, 326–334. doi:10.1097/00004703-199810000-00002

Kasari, C., Freeman, S., & Paparella, T. (2006). Joint attention and symbolic play in young children with autism: A randomized controlled intervention study. Journal of Child Psychology and Psychiatry, 47, 611–620. doi:10.1111/j.1469-7610.2005.01567.x

Kasari, C., Paparella, T., Freeman, S., & Jahromi, L. B. (2008). Language outcome in autism: Randomized comparison of joint attention and play interventions. Journal of Consulting and Clinical Psychology, 76, 125–137. doi:10.1037/0022-006X.76.1.125

Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy: methodological issues and research recommendations. Journal of Child Psychology and Psychiatry and Allied Disciplines, 44, 1116–1129. doi:10.1111/1469-7610.00195

Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883. doi:10.1001/archpsyc.59.10.877

Kravits, T. R., Kamps, D. M., Kemmerer, K., & Potucek, J. (2002). Brief report: Increasing communication skills for an elementary-aged student with autism using the Picture Exchange Communication System. Journal of Autism and Developmental Disorders, 32, 225–230. doi:10.1023/A:1015457931788

Lord, C., & McGee, J. P. (2001). Educating children with autism. Committee on educational interventions for children with autism. Division of Behavioural and Social Sciences and Education. Washington, DC: National Academy Press.

Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., . . .Rutter, M. (2000). The Autism Diagnostic Observational Schedule—Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30, 205–223. doi:10.1023/A:1005592401947

Magiati, I., & Howlin, P. (2003). A pilot evaluation study of the Picture Exchange Communication System (PECS) for children with autistic spectrum disorders. Autism, 7, 297–320. doi:10.1177/13623613030073006

McConachie, H., Randle, V., Hammal, D., & Le Couteur, A. (2005). A controlled trial of a training course for parents of children with suspected autism spectrum disorder. Journal of Pediatrics, 147, 335–340. doi:10.1016/j.jpeds.2005.03.056

Mesibov, G. B., Shea, V., & Schopler, E. (2004). The TEACCH approach to autism spectrum disorders. New York, NY: Plenum Press.

Mullen, E. M. (1995). Mullen Scales of Early Learning. Circle Pines, MN: American Guidance Service.

Pasco, G., Gordon, R. K., Howlin, P., & Charman, T. (2008). The Classroom Observation Schedule to Measure Intentional Communication (COSMIC): An observational measure of the intentional communication of children with autism in an unstructured classroom setting. Journal of Autism and Developmental Disorders, 38, 1807–1818. doi:10.1007/s10803-008-0569-3

Preston, D., & Carter, M. (2009). A review of the efficacy of the Picture Exchange Communication System intervention. Journal of Autism and Developmental Disorders, 39, 1471–1486. doi:10.1007/s10803-009-0763-y

Remington, B., Hastings, R. P., Kovshoff, H., Espinosa, F. D., Jahr, E., Brown, T., . . .Ward, N. (2007). Early intensive behavioral intervention: Outcomes for children with autism and their parents after two years. American Journal on Mental Retardation, 112, 418–438. doi:10.1352/0895-8017(2007)112[418:EIBIOF]2.0.CO;2

Rogers, S. J., & Vismara, L. A. (2008). Evidence-based comprehensive treatments for early autism. Journal of Clinical Child & Adolescent Psychology, 37, 8–38. doi:10.1080/15374410701817808

Schwartz, I. S., Garfinkle, A. N., & Bauer, J. (1998). The picture exchange communication system: Communicative outcomes for young children with disabilities. Topics in Early Childhood Special Education, 18, 144–159. doi:10.1177/027112149801800305

StataCorp. (2003). Stata statistical software(Version Release 10) [Computer software]. College Station, TX: Author.

Sulzer-Azaroff, B., Hoffman, A. O., Horton, C. B., Bondy, A., & Frost, L. (2009). The Picture Exchange Communication System (PECS): What do the data say?Focus on Autism and Other Developmental Disabilities, 24, 89–103. doi:10.1177/1088357609332743

Webb, T. (2000). Can children with autism and severe learning difficulties be taught to communicate spontaneously and effectively using the Picture Exchange Communication System. Good Autism Practice, 1, 29–41.

Yoder, P., & Compton, D. (2004). Identifying predictors of treatment response. Mental Retardation and Developmental Disabilities Research Reviews, 10, 162–168. doi:10.1002/mrdd.20013

Yoder, P., & Stone, W. L. (2006a). Randomized comparison of two communication interventions for preschoolers with autism spectrum disorders. Journal of Consulting and Clinical Psychology, 74, 426–435. doi:10.1037/0022-006X.74.3.426

Yoder, P., & Stone, W. L. (2006b). A randomized comparison of the effect of two prelinguistic communication interventions on the acquisition of spoken communication in preschoolers with ASD. Journal of Speech, Language, and Hearing Research, 49, 698–711. doi:10.1044/1092-4388(2006/051)

Yoder, P., & Warren, S. F. (1999). Maternal responsivity mediates the relationship between prelinguistic intentional communication and later language. Journal of Early Intervention, 22, 126–136. doi:10.1177/105381519902200205

Submitted: March 25, 2010 Revised: December 28, 2010 Accepted: April 20, 2011

A Meta-Intervention to Increase Completion of an HIV-Prevention Intervention: Results From a Randomized Controlled Trial in the State of Florida

Acknowledgement:

Retention and completion are critical components of the effectiveness of HIV-prevention interventions in real-world conditions and have established psychological determinants, such as attitudes and intentions (Albarracín, Durantini, Earl, Gunnoe, & Leeper, 2008; Noguchi, Albarracín, Durantini, & Glasman, 2007). Increasing retention is vital for public health because multisession behavioral interventions to reduce HIV risk are often more efficacious than single-session ones (Albarracín et al., 2005; Crepaz et al., 2014; Johnson et al., 2009; Meader et al., 2013). For example, the positive behavior change elicited by HIV-prevention interventions for clients of STI clinics is d = 0.33 for multisession programs, but only d = 0.06 for single-session programs (analyses of the data from Albarracín et al., 2005). However, when tested under conditions similar to the ones that are likely during actual implementation (e.g., lack of payments or other incentives), these multisession interventions show relatively low retention (Noguchi et al., 2007). Specifically, with the exception of interventions with captive audiences (e.g., prisons, inpatients), which show 100% completion rates, experimental interventions show a rate of completion of approximately 50% for participants initially enrolled (Branson, Peterman, Cannon, Ransom, & Zaidi, 1998; McMahon, Malow, Jennings, & Gómez, 2001). Without high retention, HIV-prevention interventions have less of an impact on behavior and clinical outcomes. Estimated associations for behavior change show that interventions with less than 50% retention rates produce a long-term decrease in HIV-safe behavior (d = −0.29), compared with an increase in HIV-safe behavior (d = 0.41) for those with 100% retention rates (Johnson et al., 2009). The present research examined the efficacy of two simple, postsession, messages to increase retention in a three-session risk-reduction counseling program. These messages were designed to either empower clients as agents responsible for their own change or highlight the instrumental outcomes of the intervention in terms of participants’ lives (e.g., addressing health concerns other than HIV, offering employment related information). The experimental design included five conditions, namely each of these messages, a combination of both, as well as two control conditions. The outcome variable was completion of a three-session counseling program.

Variability in Exposure to Behavioral Intentions

A number of interventions have been produced to change behaviors that place people at risk for HIV (Albarracín et al., 2005; Centers for Diseases Control and Prevention [CDC], 2007; Lorimer et al., 2013). These interventions are typically tested under conditions that ensure the validity of the outcome assessments (Cook & Campbell, 1979). Thus, researchers try to involve community members to see if a particular intervention works for them. Social networks are called upon to recruit these participants and numerous incentives and facilitators are used to ensure access to the desired sample of exposed participants, as well as low attrition (De Walque et al., 2012; Exner, Hoffman, Parikh, Leu, & Ehrhardt, 2002; Lauby et al., 1996; Linnan et al., 2002; Packel et al., 2012; Rabinowitz, 2002; Raj et al., 2001; Roffman, Picciano, Bolan, & Kalichman, 1997; Schilling & Sachs, 1993; Schweitzer, 1997; Tobias, Wood, & Drainoni, 2006). Although these procedures are necessary to determine whether a program works for an exposed population (efficacy trial), they remove the reluctance to participate when the intervention is implemented (Catania, Gibson, Chitwood, & Coates, 1990; Lauby et al., 1996; Packel et al., 2012). Contemporary research must thus address the fundamental scientific problem of variability in exposure to behavioral interventions, including completion of a program designed to elicit behavioral or medical change.

Despite the above-mentioned method of removing selection and attrition during tests of intervention efficacy, in real-world conditions, people choose to take part in preventive interventions (Albarracín et al., 2008; Condelli, Koch, & Fletcher, 2000; DiFranceisco et al., 1998; Hennessy, Mercier, Williams, & Arno, 2002; Noguchi et al., 2007; Rutledge, Roffman, Picciano, Kalichman, & Berghuis, 2002; Veach, Remley, Kippers, & Sorg, 2000; Wagenaar et al., 2012). Given limited time and interest, clients of health facilities can accept or refuse to take part in an HIV-prevention counseling session (Albarracín et al., 2008; Grady, Kegeles, Lund, Wolk, & Farber, 1983; Katz et al., 2015; Noguchi et al., 2007; Minder, Müller, Gillmann, Beck, & Stuck, 2002; Wilson & Albarracín, 2015). Moreover, some of the audiences most vulnerable to HIV are the least likely to complete HIV-prevention interventions (Earl et al., 2009; Liu et al., 2014; Noguchi et al., 2007; Wilson & Albarracín, 2015; Yancey, Ortega, & Kumanyika, 2006). In particular, frequent condom users are more likely to complete pro–condom-use interventions than infrequent ones (Earl et al., 2009; Noguchi et al., 2007). Thus, efficacious interventions may not reach the vulnerable audiences in need of interventions.

Given that interventions need to fully reach vulnerable audiences, not just willing ones, it is imperative to develop and test procedures that increase participation by these populations (Albarracín et al., 2008; Wilson & Albarracín, 2015). Procedures can be designed to change an audience’s behavior with respect to the preventive interventions themselves, including enrollment and retention. These procedures, termed meta-interventions, entail a standardized introduction or context change (e.g., delivery setting) intended to increase exposure to a behavioral intervention (Albarracín et al., 2008; Albarracín, Leeper, Earl, & Durantini, 2008; Wilson, Durantini, Albarracín, Crause, & Albarracín, 2013). In past research, participants with prior infrequent condom use were offered an HIV-counseling session using one of four scripted introductions to the program (Albarracín et al., 2008). A randomly assigned meta-intervention conveying that counseling participants are free not to change (empowering video) was more effective than other introductions (one promising change and another providing basic information about the counseling) or no introduction (just an offer to take part). Unobtrusive observers recorded the extent to which participants agreed to the counseling when asked, and also collected supplementary data on participants’ reading of brochures and viewing of videos. As hypothesized, the empowering meta-intervention produced high levels of enrollment in the counseling (Albarracín et al., 2008). In addition, viewing the video had an independent effect on enrollment, such that viewers of the video were more likely to enroll in counseling than nonviewers (Albarracín et al., 2008).

Selective Exposure to Interventions

Retention in HIV-prevention interventions can be understood as a form of selective exposure to information (Albarracín & Mitchell, 2004; Earl & Nisson, 2015; Noguchi et al., 2007). Selective exposure comprises biased information seeking behavior and was first studied by Festinger (1964; for reviews see Eagly & Chaiken, 1993; Freedman & Sears, 1965; Frey, 1986). Exposure to an intervention (in this case, staying in and completing a program) depends on two sets of motivations (i.e., goals or desired endstates; Lewin, 1926; see also Atkinson, 1964; McClelland, 1951; Nuttin, 1980; for other classifications of human motives, see Chaiken, Wood & Eagly, 1996; Eagly, 2007; Johnson & Eagly, 1989; Noguchi et al., 2007; Prislin & Wood, 2005; Wyer & Albarracín, 2005). On the one hand, individuals are motivated to achieve subjective self-validation, which comprises the defense of prior beliefs and practices in the domain of HIV prevention (Albarracín & Mitchell, 2004; Albarracín et al., 2008; Noguchi et al., 2007; see also Kunda, 1990; Molden & Higgins, 2005). On the other hand, individuals are motivated to maximize objective outcomes, such as reducing their risk for HIV and achieving other personal and emotional outcomes (Hart et al., 2009; Noguchi et al., 2007; Vanable et al., 2012).

A primary human motive is to achieve self-validation (Chaiken, Wood & Eagly, 1996; Eagly, 2007; Johnson & Eagly, 1989; Prislin & Wood, 2005; Wyer & Albarracín, 2005), and interventions may or may not fulfill it (Albarracín et al., 2008; Noguchi et al., 2007). Presumably due to the self-validation motive, individuals who engage in high-risk behavior are reluctant to enroll and stay in HIV-prevention interventions (Albarracín et al., 2008; Earl et al., 2009; Noguchi et al., 2007; Wilson & Albarracín, 2015).

Considering this, it is possible to design empowering messages to decrease defensiveness when recipients encounter a potential intervention that urges novel or even rejected practices (e.g., using condoms for nonusers; Albarracín et al., 2008). For example, past research has found an advantage in telling participants that change is up to them, that an intervention will simply open doors, and that they may or may not change if they participate. This type of meta-intervention puts recipients in a more active role by placing the burden of change upon them, while indirectly encouraging them to actively seek change (Amaro, 1995; Amaro & Raj, 2000; Albarracín et al., 2008; Freire, 1972; Higa, Marks, Crepaz, Liau, & Lyles, 2012; Putnam, 1911). Further, people are more likely to expose themselves to persuasive communications if they believe that they can resist their influence (Albarracín & Mitchell, 2004; Brehm, 1972; Brehm & Cohen, 1962; Watzlawick, 1978). For example, as infrequent condom users often do not want to use condoms (Albarracín, Johnson, Fishbein, & Muellerleile, 2001), highlighting the option of resistance increases their exposure to condom-use interventions (Albarracín et al., 2008). These processes have been investigated to improve enrollment in HIV programs (Albarracín et al., 2008), but not to achieve completion. As the dynamic of enrollment is similar to retention (Noguchi et al., 2007), similar messages may also increase retention in an HIV-prevention counseling program.

Besides self-validation, an important human motive is to maximize objective outcomes (Hart et al., 2009; Noguchi et al., 2007). Retention in an intervention is therefore likely to depend on the degree to which the intervention fulfills this motive (Albarracín et al., 2008; Earl et al., 2009; Noguchi et al., 2007; Vanable et al., 2012). For HIV-risk reduction interventions, objective outcomes include HIV-risk reduction (Floyd, Prentice-Dunn, & Rogers, 2000; Rosenstock, Strecher, & Becker, 1994), but also emotional and instrumental support (Durantini & Albarracín, 2009; Vanable et al., 2012). For people who engage in high-risk behavior, the risk-reduction outcome can conflict with the self-validation motive (Albarracín et al., 2008; Earl et al., 2009; Noguchi et al., 2007). Thus, emphasizing that the objective of an intervention is to change participants’ risk behavior can lead participants to reject the intervention and feel manipulated (Albarracín et al., 2008). In contrast, emphasizing the emotional, social and instrumental value of an intervention beyond HIV prevention can entice participation (Durantini & Albarracín, 2007, 2009; Liu et al., 2014). Past research supports this assertion, showing that, women seek out programs that provide social and emotional support (i.e., company, encouragement, and affection), whereas men seek out programs that provide instrumental support (i.e., health care or payments; Durantini & Albarracín, 2007, 2009). In this light, we tested the effect of messages that emphasize the emotional, instrumental, and (non-HIV related) physical health outcomes of returning to the sessions of an HIV-prevention and counseling program.

At the end of 2012, an estimated 1,218,400 people in the United States were living with HIV/AIDS (Hall et al., 2015). HIV incidence has remained relatively stable since the mid-1990s, with an estimated 50,000 persons becoming infected with HIV on any given year (Hall et al., 2008). Based on confidential-name–based HIV reports, 47,352 cases of HIV/AIDS were diagnosed in 35 U.S. areas (33 states, Guam, and the U.S. Virgin Islands) in 2013. Up to 2012, the cumulative number of individuals dead by HIV was 658,507, with Florida having one of the highest HIV disease death rates in the U.S. (Florida Department of Health [FDH], 2012). Also, in 2013, Florida ranked first in new infections per year (5,377 new infections) and second in number of cumulative reported HIV cases (49,058; CDC, 2013). In 2014, the largest estimated proportion of HIV/AIDS diagnoses in Florida was for men who have sex with men (MSM), and ethnic minority adults and adolescents infected through heterosexual contact (FDH, 2015). Clearly, protecting Floridians is a national health priority, particularly those from African American backgrounds, who are highly represented in our population.

In Florida, prevention is the most important tool to avoid an even more accelerated epidemic (FDH, 2007). Duval County (Electoral district 4) is an important area that has received relatively little research attention (compared with Dade County, e.g.). Considering sheer number of cases in 2014, Duval County, which includes Jacksonville, ranks 1st for Gonorrhea and 4th for Chlamydia (FDH, 2014a), and 4th for HIV (FDH, 2014b). Of 67 counties in the state of Florida, these rankings place the region at very high risk. Given these findings, ensuring intervention effectiveness for this population is key.

A randomized control trial was used to test the impact of video meta-interventions designed to either empower clients or remind them of the various objective goals fulfilled by the HIV-counseling program, and to compare these videos with control videos. The empowering meta-intervention entailed presenting the recipient as the motor of the behavior change (Albarracín et al., 2008). This strategy emphasized that the program could not change behavior unless the individual wanted it to. The instrumental video included descriptions of the sort of information and referrals the counselor could provide, in addition to giving information and guidance about HIV prevention. There was also a condition that combined the empowering and instrumental messages, as well as two control conditions. One control included stories about people living with HIV that were used in all experimental videos. The other control was more minimal, and simply presented educational information on reducing STIs. Thus, the design comprised five conditions to analyze their impact on completion of a CDC-recommended, three-session counseling program.

Our design also included a factor signaling whether perceptions of the video were measured. Although it was important to attempt to measure whether the videos induced expectations of empowerment and instrumental outcomes, including such blunt measures often alters the outcomes of experimental designs (Dholakia & Morwitz, 2002; Morwitz, Johnson, & Schmittlein, 1993). As a compromise, we randomized whether these measures appeared, and so only half of the sample completed these measures immediately after watching the meta-intervention video.

Enrollment

Clients from the STI clinics from the Florida Department of Health in Duval County were recruited (via flyers, referrals) for a study testing a three-session counseling program. To be eligible, individuals had to be between 18 and 35 years of age, report engaging in sexual activity in the past three months, and report using condoms never or occasionally. Participants were excluded if they were HIV-positive, or were trying to get pregnant or had a partner who was trying to get pregnant. Eligible participants were scheduled for their first study appointment. To ensure initial enrollment, participants were paid $35 for attending the first session, and $15 for subsequent sessions. The study was approved by the Institutional Review Boards (IRBs) of the University of Illinois, University of Pennsylvania, and Florida Department of Health, and each participant provided informed consent. Figure 1 describes all exclusions and Ns resulting from assignment procedures. The maximal control was by design smaller than the other conditions, including the minimal control. The trial was preregistered in clinicaltrials.gov (NCT01152281).

Figure 1. Recruitment and assignment.

Participants

Seven hundred twenty-two eligible participants (58% female) attended the initial counseling session, with a retention rate of 76% for the second session and a completion rate of 63% at the third session. The mean age of the sample was 26.54 (SD = 4.78). The majority of participants were African American (79%), and generally had an income under $9,999 per year (58%). Eighty-five percent reported having a main partner with whom they had a relationship on average of 2.37 years (SD = 2.10). Condom use in this sample was low, with only 1.1% reporting always using a condom when they had sex with their main partner. A full description of the same appears in Table 1.

Sample Description

The Counseling Intervention

The model of counseling that was used entailed a client-centered, cost-effective HIV-prevention program (CDC, 1993, 2007; Holtgrave, Valdiserri, Gerber, & Hinman, 1993; Kamb et al., 1998) facilitated by a counselor. This model’s efficacy has been demonstrated to significantly reduce STIs in a large multisite study (Kamb et al., 1998) and continues to be recommended as a standard for one-on-one counseling (CDC, 2007). The counseling seeks to reduce HIV risk behaviors by giving information, identifying risk behaviors, as well as steps to change them, and developing behavioral skills enabling safer behavior. This counseling can involve one or more sessions lasting at least 20 min, all of them following the same format. In our proposed study, a three-session model was used.

During the first session, participants received information regarding HIV transmission and prevention tailored to their culture, language, sex, gender, age, and educational level. The counselor ensured that the participant understood the information and that all of their misconceptions were corrected. The participant was encouraged to ask questions and clear their doubts. Following the informative part of the session, the counselor performed a personalized risk assessment, encouraging the participant to identify, understand, and acknowledge the behaviors and circumstances that put them at risk for being infected by HIV. Addressed topics included factors associated with risk behavior, such as using drugs or alcohol before sex, underestimating personal risk, having low self-efficacy, having distorted or fatalistic beliefs, and misperceiving peer norms. In addition, the counselor examined previous attempts made by the participant to reduce their risk and identified the reasons for their success or failure in these situations. This in-depth exploration allowed the counselor to help the participant consider ways to reduce personal risk and commit to a single, reachable step toward change. Once this risk assessment was complete, the counselor asked the participant to describe the risk-reduction step to be attempted (while acknowledging positive steps made), and helped the participant identify and commit to additional behavioral steps. Testing was also discussed, with referrals provided as needed.

During the following sessions, the counselor and the participant explored the success or failure of the steps proposed, and adjusted goals to the participant’s achievements. Furthermore, the second and third sessions also included a module for providing emotional support and addressing instrumental and/or medical concerns, in addition to HIV. This inclusion fulfilled the goal of providing supporting objective outcomes highlighted in some of the meta-intervention conditions. Among other things, after the HIV-risk reduction portion of the second and third counseling session was complete, the counselor discussed the physical and psychological symptoms, made referrals and provided information. This modification allowed us to test the effectiveness of messages that emphasized emotional and physical outcomes, with the counseling providing some venue for relief.

The counselors had good fidelity ratings using standard observation lists, and great high on cultural competency as measured with a valid and reliable questionnaire (Ponterotto, Alexander, & Grieger, 1995; Ponterotto, Potere, & Johansen, 2002). The counselors used written guides and records to ensure the use of a standardized procedure, and were closely supervised and retrained periodically. Therefore, after initial intensive training before the trial began, a check of videotaped sessions was performed to ensure proper application of the protocol. A random sample of 38 sessions showed 100% adherence to seven key dimensions of the protocol, which included appropriate introduction to the session, adequate performance of the risk assessment, proper evaluation of personal resources, proper evaluation of barriers, adequate integration of personal resources into newly set goals, and a clear closure. The average duration of the sessions was 25 min.

Meta-Intervention Messages

The messages were 24- to 34-min videos presented at the end of the first counseling session, to infer effects on retention at the second and third sessions. There were five videos, one for each condition, one resulting from crossing two meta-interventions, and two control videos.

The first experimental video, lasting 28 min, presented a meta-intervention conveying the message of being empowered (empowering condition). The video presented community members who talked about their experiences with HIV and counseling. This content was interspersed with messages delivered by these characters and professionals, conveying that subsequent counseling sessions were not intended to force change upon individuals. The stories were set at local places in North Florida (e.g., a fishing environment, a bar) with local music used in the background. The videos contained material in both Spanish and English, subtitled to the other language.

The second experimental video, lasting 26 min, presented a meta-intervention emphasizing the objective outcomes associated with HIV-prevention counseling (instrumental condition). This video presented the same stories as the first message. However, this experimental video also emphasized the emotional, social and objective (i.e., non-HIV/STI health) outcomes of returning to the counseling sessions. In this message, characters and professionals described how HIV-prevention counseling was also a venue to discuss personal problems, such as violence in the home or depression, and the extent to which many clients find emotional relief and social support from participating in the counseling. This message thus conveyed how HIV-prevention counseling often facilitates the treatment of other health problems, while also providing a venue for obtaining information about, and referral to, social services.

The third experimental video, lasting approximately 34 min, combined the first and the second meta-intervention messages. The final two videos were both control videos. The first, which lasted approximately 26 min, included the same stories and locations as those presented in the other three videos, but did not contain any of the meta-intervention messages (minimal control condition). The second, lasting 24 min, contained neither stories presented by local characters, nor any meta-intervention messages, but simply presented short vignettes aimed at increasing behavioral skills, perceived risk and knowledge about reducing STIs (maximal control condition; selected from video developed by Warner et al., 2008).

The use of two control conditions let us disentangle the effects of the meta-intervention messages from the community stories about HIV and counseling. Specifically, the difference between the minimal control video and those used in the three experimental conditions was the absence of a meta-intervention message; the rest of the content (e.g., the community stories) remained the same. The maximal control did not have either, and only presented risk and facts about STIs. Thus, it became possible to see whether differences in the three experimental conditions were attributable to the combination of the meta-intervention message and stories that were included, or were exclusive to the meta-intervention message. This condition was added after the project was funded and therefore had to be smaller because of funding constraints.

Design

This study crossed two meta-interventions: (a) a video message empowering or validating the client to return to the sessions, and (b) a video message highlighting opportunities for emotional and instrumental support (e.g., information about cardiovascular health, referrals etc.) facilitated by HIV-prevention counseling. The design had another factor, which concerned the inclusion of measures of expectation induced by the video, which were to appear immediately following the video. Only half of the sample completed these measures with the objective of avoiding measurement sensitivity. Thus, our design was a 2 (empowering meta-intervention: present vs. absent) × 2 (instrumental meta-intervention: present vs. absent) × 2 (measurement of mediators: present or absent) between-subjects factorial with the addition of a minimal control condition.

Baseline Measures

Data was collected using audio computer-assisted self-interview (ACASI) procedures. With this technique, participants listened to the question, while simultaneously reading them on the screen. ACASI procedures have been reported to increase accuracy with respect to non-normative behaviors and responses, thus decreasing the effects of social desirability and experimental demand (see, e.g., Des Jarlais et al., 1999; Mensch, Hewett, & Erulkar, 2003; Williams et al., 2000). Questionnaires were available in Spanish for participants who preferred it.

Baseline questionnaires measuring past behavior, intentions, and demographics were collected from participants before the start of their first counseling session. Questionnaire items were first transformed to a z-score, and then averaged, to produce a composite measure of condom, drug, alcohol and injection drug use, as well as a composite measure for number of sexual partners and intentions to use condoms.

Condom use and unprotected sex

Participants were asked about their condom use during intercourse in (a) the past month, (b) the past three months and (c) the past six months, as well as (d) how often they use condoms in general, (e) how many times they engaged in unprotected sex in the past six months, and (f) whether they used a condom the last time they had intercourse. These questions were asked in reference to participants’ main and other partner(s), and had acceptable internal consistency (α = .69 for main partner, and α = .63 for other partner).

Number of sexual partners

Participants were asked about the number of sexual partners they had in (a) the past 48 hours, (b) the past month, and (c) the past six months. This measure (National Institute of Drug Abuse [NIDA], 1991, 1993) had good internal consistency in our sample (α = .77; see also Edwards, Fisher, Johnson, Reynolds, & Redpath, 2007; Needle et al., 1995).

Alcohol use

Participants were asked to report their behaviors related to prior alcohol use. For those participants who reported that they drink alcohol, alcohol-use consisted of a single-item measure including reports of the number of times participants used alcohol during the past week.

Drug use

Participants were also asked to report their behaviors related to prior drug use. Drug-use measures included reports of the number of times participants used drugs (in general, as well as heroin, crack, and cocaine) during (a) the past 48 hours and (b) the past month. This measure had poor internal consistency in our sample (α = .52).

Injection drug use

Injection drug use was differentiated from the broader measure of drug use, as the level of HIV risk conferred by intravenous drug users is higher. Participants were asked (a) the number of times they injected drugs, (b) the frequency of sharing syringes, (c) the number of sharing partners, and (d) the number of times the equipment was sterilized between uses over a period of the past 48 hours, past month, and past six months. These questions were validated against HIV infection rates by Anthony et al. (1991), and had good internal consistency in our study (α = .85).

Intentions to use condoms

Participants were also asked about their intentions to use condoms, using previously validated measures (Albarracín et al., 2000; Earl et al., 2009; Kamb et al., 1998). Specifically, participants were asked how likely it was for them to use a condom with their partner (a) the next time they had intercourse, (b) every time for the next three months they had intercourse, and (c) every time for the next six months they had intercourse. Participants were also asked about (d) the strength of their intentions and (e) their motivation to use condoms with their partner in the next six months. These questions were asked in reference to participants’ main and other partner(s), and had excellent internal consistency (α = .94 for main partner, and α = .96 for other partner).

In addition to the above measures, participants were also asked standard items from the General Social Survey (http://gss.norc.org/) to assess structural variables, namely household income, level of education, race/ethnicity, and employment.

Measures of Video Acceptability, Counseling Expectations, and Return Intentions

The design included measures of the acceptability of the video, expectations of the following counseling sessions, and intentions to return. Measures were completed after the presentation of the video, by only half of the participants. Items for each measure were first transformed to a z-score, and then averaged, to create a composite measure for video acceptability, counseling expectations and return intentions.

To gauge video acceptability, participants were asked whether the video presented was (a) interesting, (b) useful, (c) enjoyable, (d) clear, and (e) relevant, as well as whether the video (f) made participants think, (g) taught them about condom use, and (h) presented new information. Participants were also asked whether the video made them (i) nervous, (j) worry, (k) feel compelled to do something they did not want to do, and (l) feel forced to change their beliefs or behaviors. Items (i) to (l) were first reverse scored, and then averaged with items (a) to (h). These measures had high internal consistency (α = .80).

Participants were also given measures of expectations about the counseling. Specifically, we asked participants whether they thought that the counseling would (a) force or (b) compel them to do things they did not like, (c) make them do things to please the counselor, (d) increase HIV safe behavior, (e) help them discuss health problems besides HIV and STIs, and (f) help them with their emotional concerns. Items (a) through (c) addressed empowerment expectations (α = .65), and items (d) through (f) addressed instrumental outcomes (α = .76).

Finally, participants were asked about the (a) strength of their return intentions and (b) how much they would enjoy returning (α = .67). All these measures were included as potential process data.

Completion Measure

Retention was observed during the last two sessions. When participants started the first session, the counselor indicated that the complete counseling program included two additional follow-up sessions. We measured retention by taking into account whether the participant completed all three sessions.

Across the board, there was a high completion rate of 63%, which is probably attributable to the use of payments for attendance at the return sessions. Before analyzing the outcome of the meta-intervention, we compared the demographic and behavioral profile of our sample. Any incidental difference was then controlled for in the main analysis.

Comparability Across Conditions

Although random assignment was intended to ensure comparability across conditions, we performed periodic checks to make sure there were no gender, age or race biases in the participant distribution. Table 1 provides a summary of relevant sample characteristics, by overall sample, as well as broken down by condition. One-way ANOVAs and chi-square tests revealed no significant difference in these variables across our five conditions (ps > .077), with the exception of age and race. Variability in race across conditions approached significance, χ(18) = 28.71, p = .052. There was a significant difference in the age of participants across conditions, F(9, 712) = 2.15, p = .024. A Tukey post hoc test revealed that participants’ age was significantly lower in the combined instrumental and empowering meta-intervention condition, when no measures of return expectations and intentions were included (M = 25.08, SD = 4.46), compared with the same condition presented when those variables were measured (M = 27.52, SD = 5.06). There were no significant differences in age across the other conditions (ps > .089).

Effects on Video Acceptability, Counseling Expectations, and Return Intentions

A multivariate analysis of variance, with our five meta-intervention message conditions as a factor, was conducted to analyze the impact of our experimental factors on video acceptability, counseling expectations (either empowering or instrumental) and the intention to return to counseling. Results revealed no significant effect of meta-intervention message (p = .14), indicating that our experimental factor did not affect participants’ reported acceptability of the video, empowering or instrumental expectations of counseling, or their intentions to return to the next counseling session. These findings suggest that any effect of the video either occurred outside of awareness, or could not be clearly reported by our participants on the scales that we developed. Means and standard deviations for video acceptability, counseling expectations and return intentions are presented in Table 2. The means in all cases were above the midpoints of the scales and suggest favorable perceptions of the video and a program perceived to be acceptable.

Means and Standard Deviations for Video Acceptability, Counseling Expectations, and Return Intentions Presented Across Meta-Intervention Message Conditions

Main Experimental Results

A logistic regression analysis with our three factors of empowering message, instrumental message, and mediator measurement presence, as well as all interactions, was conducted to analyze the impact of our experimental factors on counseling completion. In this analysis, the two control conditions were combined but a separate consideration of these two conditions does not alter our results. The analyses entailed a forward removal of predictors. The results from this analysis appear in Table 3. Results revealed a significant two-way interaction between the presentation of instrumental and empowering messages, as well as a significant interaction between mediator measure presence and presentation of instrumental messages (see Figure 2). Results indicated that the instrumental message alone was better than any of the other messages. Furthermore, the instrumental message was more effective than the empowering message in the absence of measures of mediating expectations. The overall gains in completion as a result of the instrumental message were 16%, suggesting success in the intended facilitation of counseling completion.

Final Results From Logistic Regression Analysis

Figure 2. Effects of meta-interventions.

This paper reported a large and complex randomized controlled trial testing meta-interventions to increase completion of a CDC-recommended counseling for HIV prevention. Our results identified a successful program—one that incorporates the counseling within a broader spectrum of goals that are likely salient to the clients of most programs to prevent, test for, and treat HIV. This finding is particularly impressive given that the completion rates in the sample were fairly high, probably attributable to a combination of excellent counseling technique and highly effective counselors, in addition to the use of payments for follow-up sessions. In other words, the room for improvement may have been limited to begin with, or at least, limited relative to the usually lower completion rates in the average comparable HIV-prevention trial (see Albarracín et al., 2005). Also, although the use of payments seemed desirable given pilot data showing lower completion than that ultimately obtained, hindsight suggests that the payments might have reduced the sensitivity of our completion measure.

The instrumental meta-intervention seemed important to test as it involves a patient centered approach to interventions (Lauver et al., 2002; Morgan & Yoder, 2012; Robinson, Callister, Berry, & Dearing, 2008) that is entirely consistent with psychological theories of persuasion and motivation. For a message to be well received, it is necessary for its content to be relevant to the audience and sufficiently consistent to ensure high level of message-consistent thinking and low levels of counterarguing (Albarracín, Johnson, & Zanna, 2005; Albarracín & Vargas, 2010). In the case of the instrumental message, highlighting the various personal goals that can be met through contact with the health system and associated services clearly retained participants who otherwise may have dropped out from the program. Future research should be conducted to replicate this finding in other areas, particularly HIV testing, HIV treatment, and introduction of pharmacological agents, such as PrEP.

Three aspects of our findings are noteworthy. First, the empowering meta-intervention, which had impressive results in a trial to increase acceptance of HIV-prevention counseling (Albarracín et al., 2008), did not yield improved completion. This result highlights that the determinants of enrollment and retention are different, with defensiveness playing a key role in initiation, but lack of relevance or perceived purpose probably underlying drop out. Second, the average completion rate was rather high and so our meta-intervention may have stronger effects when completion is low to begin. In our case, the high quality of the counseling and intensive training and supervision of the counselors, along with the payments, decreased the need for an intervention to ensure completion. Replications in conditions that are more conducive to higher drop out will therefore be highly informative. Third, as is common in testing behavioral interventions, the mediation analysis shed no light on the variables that led to the treatment outcome. It is of course possible that expectations did change, but participants did not have full introspective access to these contents due to the operation of relatively nonconscious processes. More likely, however, the questions were too involved and required a level of metacognition that is unfortunately not frequent for a sample with a low level of education. Perhaps a less directive assessment, such as a qualitative interview, may in the future increase understanding of the reasons underlying the success of the instrumental message.

Effects of the Mediators

The introduction of the mediators was expected to affect completion by sensitizing clients to the importance of completion. Often, calling attention to what the goals of a study are can introduce demands effects (Barabasz & Barabasz, 1992). An interesting study on measurement effects, however, was conducted by Glasman and colleagues (2015), who found that introducing measurements of risky behavior decreased the effect of a prevention intervention, suggesting a potential underestimation of the effect of behavioral programs. Both the demand and efficacy reduction patterns are entirely consistent with what we found. The inclusion of mediator measures increased completion, while also decreasing sensitivity to the meta-intervention. It seems likely that in-depth questions elicit cognitive and motivational processes, such as self-talk, that distract recipients from fully processing messages received immediately before (Glasman et al., 2015) or after (in our study) receiving a persuasive communication.

Remaining Questions and Limitations

There are several important questions to address, including possible differences in the intervention as a function of the delivered meta-intervention. For all clients, during the second and third counseling sessions, counselors discussed the physical and psychological symptoms associated with HIV, addressed instrumental and medical concerns, and provided emotional support for the participants regardless of the condition they were randomized to. Thus, concern over the influence of a meta-intervention condition on counselors’ interactions with participants is mitigated by the fact that the delivery of the counseling and the delivery of the meta-intervention messages were done by different team members. The counselor was blind to the meta-intervention condition, and so, all subsequent interactions with participants could not have been biased by knowledge of experimental condition.

With respect to generalizability to the population, we restricted the sample of participants to 18- to 35-year-olds because the estimated number of diagnoses of HIV infections in the U.S. is highest for this age range (Center for Disease Control, 2014). Additionally, prior work has shown that the mean age of participants enrolling in HIV-prevention intervention programs falls within this range (e.g., Liu et al., 2014; Wilson, Durantini, Albarracín, Crause, & Albarracín, 2013). Despite this age range restriction, we do not believe the generalizability of the studies should be affected, as the meta-interventions used target broad psychological themes, such as seeking self-validation or maximizing objective outcomes, which are not limited to specific age groups.

With respect to generalizability to the intervention format, a three-session counseling program was selected both because previous work we have done showed three-session interventions were a good length (Liu et al., 2014), and because of cost considerations. In principle, it is possible that the meta-intervention messages used in this study might have different efficacy with a longer program. However, given that completion was very high to begin, a program with lower rates of completion may show a stronger effect of the meta-interventions. Furthermore, we limited the presentation of the meta-intervention messages to participants who attended the first session as we were interested in the effect of these messages on completion of a program after it starts. Prior work we have conducted has already addressed the benefits of certain meta-intervention messages on increasing enrollment in HIV-prevention intervention programs (Albarracín et al., 2008), where it made more sense to present these messages to the entire sample at baseline.

One important consideration is the potential effect of the financial incentive to participation used in this study. As is well known, paying individuals for performing a task can reduce the perception of freedom of choice, and in turn decrease intrinsic motivation for the task (see, e.g., Festinger, 1964). In this context, payments could have led to lesser motivation to complete the program than lack of payments. This possibility seems unlikely because of the very high completion rates we obtained in our study. A likely possibility, however, is that the payment might have decreased motivation for the empowering condition, which emphasized freedom of choice. For example, emphasizing that clients are active participants and the motor of change may have reminded them of the payment and thus reduce their motivation to complete the program. Thus, future work should test the meta-intervention in the absence of payments.

It is important to further consider the effect of the meta-interventions, particularly the fact that the empowering one seemed to offer no benefits. In this regard, although empowering messages have been effective at eliciting enrollment before the intervention is delivered, a retention meta-intervention of this type may be directly in conflict with the obvious behavior-change intent of the program. The preventive nature of the program is likely to be apparent from a first session in which participants are encouraged to identify, understand and acknowledge the behaviors and circumstances that put them at risk for being infected with HIV. Also, it seems possible that the video might become impractical in some contexts, particularly if it were excessively long. However, the counseling session lasted 25 min in average, which along with a 26-min instrumental video would result in a first session of 51 min of length. Thus, it seems possible to integrate this meta-intervention into the regular operation of a clinic.

Although efficacious interventions are key tools in the prevention, detection, and treatment of HIV, the public health impact of these interventions are likely reduced when vulnerable populations do not complete the program. Our findings provide evidence that a meta-intervention simply describing HIV-prevention counseling as a venue where one can discuss personal problems or medical needs, and receive appropriate referrals to community resources, appears to be a promising strategy for increasing retention. The development of cost-effective tools to retain clients in multisession HIV-prevention programs could have a significant impact on the lives of those at greatest risk for HIV infection and may play a pivotal role in decreasing the number of new HIV infections in Florida, and in the Nation.

¹ Only one participant asked for the Spanish version of the measures.

Albarracín, D., Durantini, M. R., Earl, A., Gunnoe, J. B., & Leeper, J. (2008). Beyond the most willing audiences: A meta-intervention to increase exposure to HIV-prevention programs by vulnerable populations. Health Psychology, 27, 638–644. 10.1037/0278-6133.27.5.638

Albarracín, D., Gillette, J. C., Earl, A. N., Glasman, L. R., Durantini, M. R., & Ho, M. H. (2005). A test of major assumptions about behavior change: A comprehensive look at the effects of passive and active HIV-prevention interventions since the beginning of the epidemic. Psychological Bulletin, 131, 856–897.

Albarracín, D., Ho, R. M., McNatt, P. S., Williams, W. R., Rhodes, F., Malotte, C. K., . . . the Project RESPECT Study Group. (2000). Structure of outcome beliefs in condom use. Health Psychology, 19, 458–468. 10.1037/0278-6133.19.5.458

Albarracín, D., Johnson, B. T., Fishbein, M., & Muellerleile, P. A. (2001). Theories of reasoned action and planned behavior as models of condom use: A meta-analysis. Psychological Bulletin, 127, 142–161. 10.1037/0033-2909.127.1.142

Albarracín, D., Johnson, B. T., & Zanna, M. P. (2005). Attitudes: Introduction and scope. In D.Albarracín, B. T.Johnson, & M. P.Zanna (Eds.), The handbook of attitudes (pp. 3–20). Hillsdale, NJ: Erlbaum.

Albarracín, D., Leeper, J., Earl, A., & Durantini, M. R. (2008). From brochures to videos to counseling: Exposure to HIV-prevention programs. AIDS and Behavior, 12, 354–362. 10.1007/s10461-007-9320-x

Albarracín, D., & Mitchell, A. L. (2004). The role of defensive confidence in preference for proattitudinal information: How believing that one is strong can sometimes be a defensive weakness. Personality and Social Psychology Bulletin, 30, 1565–1584. 10.1177/0146167204271180

Albarracín, D., & Vargas, P. (2010). Attitudes and persuasion: From biology to social responses to persuasive intent. In S. T.Fiske, D. T.Gilbert, & G.Lindzey (Eds.), The handbook of social psychology (pp. 394–427). Hoboken, NJ: Wiley. 10.1002/9780470561119.socpsy001011

Amaro, H. (1995). Love, sex, and power. Considering women’s realities in HIV prevention. American Psychologist, 50, 437–447. 10.1037/0003-066X.50.6.437

Amaro, H., & Raj, A. (2000). On the margin: Power and women’s HIV risk reduction strategies. Sex Roles, 42, 723–749. 10.1023/A:1007059708789

Anthony, J. C., Vlahov, D., Celentano, D. D., Menon, A. S., Margolick, J. B., Cohn, S., . . .Polk, B. F. (1991). Self-report interview data for a study of HIV-1 infection among intravenous drug users: Description of methods and preliminary evidence on validity. Journal of Drug Issues, 21, 739–757. 10.1177/002204269102100405

Atkinson, J. W. (1964). An introduction to motivation. Princeton, NJ: D. Van Nostrand.

Barabasz, A. F., & Barabasz, M. (1992). Research designs and considerations. In E.Frornm & M. R.Nash (Eds.), Contemporary hypnosis research (pp. 173–200). New York, NY: Guilford Press. The preceding paper attributes the concept to Weber, S. J., & Cook, T. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77, 273–295.

Branson, B. M., Peterman, T. A., Cannon, R. O., Ransom, R., & Zaidi, A. A. (1998). Group counseling to prevent sexually transmitted disease and HIV: A randomized controlled trial. Sexually Transmitted Diseases, 25, 553–560. 10.1097/00007435-199811000-00011

Brehm, J. W. (1972). Responses to loss of freedom: A theory of psychological reactance. Morristown, NJ: General Learning Press.

Brehm, J. W., & Cohen, A. R. (1962). Explorations in cognitive dissonance. New York, NY: Wiley. 10.1037/11622-000

Catania, J. A., Gibson, D. R., Chitwood, D. D., & Coates, T. J. (1990). Methodological problems in AIDS behavioral research: Influences on measurement error and participation bias in studies of sexual behavior. Psychological Bulletin, 108, 339–362. 10.1037/0033-2909.108.3.339

Centers for Disease Control and Prevention. (2007). Revised guidelines for HIV counseling, testing, and referral. Morbidity Mortality Weekly Report, 50, 1–58.

Centers for Disease Control and Prevention. (2013). HIV surveillance report (Vol. 25). Atlanta, GA: U.S. Department of Health and Human Services.

Centers for Disease Control and Prevention. (2014). HIV surveillance report (Vol. 26). Atlanta, GA: U.S. Department of Health and Human Services.

Centers for Disease Control and Prevention. (1993). Update: Barrier protection against HIV infection and other sexually transmitted diseases. Morbidity Mortality Weekly Report, 42, 589–591, 597.

Chaiken, S., Wood, W., & Eagly, A. H. (1996). Principles of persuasion. In E. T.Higgins & A. W.Kruglanski (Eds.), Social psychology: Handbook of basic principles (pp. 702–742). New York, NY: Guilford Press.

Condelli, W. S., Koch, M. A., & Fletcher, B. (2000). Treatment refusal/attrition among adults randomly assigned to programs at a drug treatment campus: The New Jersey Substance Abuse Treatment Campus, Seacaucus, NJ. Journal of Substance Abuse Treatment, 18, 395–407. 10.1016/S0740-5472(99)00086-0

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin Company.

Crepaz, N., Tungol-Ashmon, M. V., Higa, D. H., Vosburgh, W., Mullins, M. M., Barham, T., . . .Lyles, C. M. (2014). A systematic review of interventions for reducing HIV risk behaviors among people living with HIV in the United States, 1988–2012. AIDS, 28, 633–656. 10.1097/QAD.0000000000000108

Des Jarlais, D. C., Paone, D., Milliken, J., Turner, C. F., Miller, H., Gribble, J., . . .Friedman, S. R. (1999). Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: A quasi-randomised trial. Lancet, 353, 1657–1661. 10.1016/S0140-6736(98)07026-3

de Walque, D., Dow, W. H., Nathan, R., Abdul, R., Abilahi, F., Gong, E., . . .Medlin, C. A. (2012). Incentivising safe sex: A randomised trial of conditional cash transfers for HIV and sexually transmitted infection prevention in rural Tanzania. British Medical Journal Open, 2, e000747. 10.1136/bmjopen-2011-000747

Dholakia, U. M., & Morwitz, V. G. (2002). The scope and persistence of mere-measurement effects: Evidence from a field study of consumer satisfaction measurement. Journal of Consumer Research, 29, 159–167. 10.1086/341568

DiFranceisco, W., Kelly, J. A., Sikkema, K. J., Somlai, A. M., Murphy, D. A., & Stevenson, L. Y. (1998). Differences between completers and early dropouts from 2 HIV intervention trials: A health belief approach to understanding prevention program attrition. American Journal of Public Health, 88, 1068–1073. 10.2105/AJPH.88.7.1068

Durantini, M. R., & Albarracín, D. (2007). Emotional and physical concerns as motives for seeking HIV prevention counseling. Unpublished manuscript.

Durantini, M. R., & Albarracín, D. (2009). Material and social incentives to participation in behavioral interventions: A meta-analysis of gender disparities in enrollment and retention in experimental human immunodeficiency virus prevention interventions. Health Psychology, 28, 631–640. 10.1037/a0015200

Eagly, A. H. (2007). In defense of ourselves: The effects of defensive processing on attitudinal phenomena. In M.Stroebe, J.de Wite, M.Hewstone, K.van den Bos, & H.Schut (Eds.), The scope of social psychology: Theory and applications (pp. 65–83). London, UK: Psychology Press.

Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. Fort Worth, TX: Harcourt Brace Jovanovich.

Earl, A., Albarracín, D., Durantini, M. R., Gunnoe, J. B., Leeper, J., & Levitt, J. H. (2009). Participation in counseling programs: High-risk participants are reluctant to accept HIV-prevention counseling. Journal of Consulting and Clinical Psychology, 77, 668–679. 10.1037/a0015763

Earl, A., & Nisson, C. (2015). Applications of selective exposure and attention to information for understanding health and health disparities. Emerging Trends in the Social and Behavioral Sciences: An Interdisciplinary, Searchable, and Linkable Resource, 1–14. 10.1002/9781118900772.etrds0013

Edwards, J. W., Fisher, D. G., Johnson, M. E., Reynolds, G. L., & Redpath, D. P. (2007). Test-retest reliability of self-reported drug treatment variables. Journal of Substance Abuse Treatment, 33, 7–11. 10.1016/j.jsat.2006.11.007

Exner, T. M., Hoffman, S., Parikh, K., Leu, C. S., & Ehrhardt, A. (2002). HIV counseling and testing: Women’s experiences and the perceived role of testing as a prevention strategy. Perspectives on Sexual and Reproductive Health, 34, 76–83. 10.2307/3030210

Festinger, L. (1964). Conflict, decision, and dissonance. Stanford, CA: Stanford University Press.

Florida Department of Health. (2007). The Florida Division of Disease Control Surveillance Report. Tallahassee, FL: Author.

Florida Department of Health. (2012). HIV mortality in Florida in 2012. Tallahassee, FL: Bureau of Vital Statistics and Bureau of Communicable Diseases, HIV/AIDS Section.

Florida Department of Health. (2014a). Bureau of STD Prevention & Control. Retrieved from http://www.floridahealth.gov/diseases-and-conditions/disease-reporting-and-management/disease-reporting-and-surveillance/data-and-publications/_documents/2014-fl-msr.pdf#search=%222014%20gono%20chla%20bureau%20of%20std%20prevention%20%20%20control%20surveillance%22

Florida Department of Health. (2014b). Bureau of HIV/AIDS. Retrieved from http://www.floridahealth.gov/diseases-and-conditions/aids/surveillance/index.htmla

Florida Department of Health. (2015). HIV Disease: U.S. vs. Florida. Tallahassee, FL: Bureau of Vital Statistics and Bureau of Communicable Diseases, HIV/AIDS Section.

Floyd, D. L., Prentice-Dunn, S., & Rogers, R. W. (2000). A meta-analysis of research on protection motivation theory. Journal of Applied Social Psychology, 30, 407–429. 10.1111/j.1559-1816.2000.tb02323.x

Freedman, J. L., & Sears, D. O. (1965). Selective exposure. In L.Berkowitz (Ed.), Advances in experimental social psychology (Vol. 2, pp. 58–97). New York, NY: Academic Press.

Freire, P. (1972). Pedagogy of the oppressed. Harmondsworth, UK: Penguin.

Frey, D. (1986). Recent research on selective exposure to information. In L.Berkowitz (Ed.), Advances in experimental social psychology (Vol. 19, pp. 41–80). San Diego, CA: Academic Press. 10.1016/S0065-2601(08)60212-9

Glasman, L. R., Skinner, D., Bogart, L. M., Kalichman, S. C., McAuliffe, T., Sitzler, C. A., . . .Weinhardt, L. S. (2015). Do assessments of HIV risk behaviors change behaviors and prevention intervention efficacy? An experimental examination of the influence of type of assessment and risk perceptions. Annals of Behavioral Medicine, 49, 358–370. 10.1007/s12160-014-9659-y

Grady, K. E., Kegeles, S. S., Lund, A. K., Wolk, C. H., & Farber, N. J. (1983). Who volunteers for a breast self-examination program? Evaluating the bases for self-selection. Health Education Quarterly, 10, 79–94. 10.1177/109019818301000201

Hall, H. I., An, Q., Tang, T., Song, R., Chen, M., Green, T., . . . the Centers for Disease Control and Prevention (CDC). (2015). Prevalence of diagnosed and undiagnosed HIV infection—United States, 2008–2012. Morbidity Mortality Weekly Report, 64, 657–662.

Hall, H. I., Song, R., Rhodes, P., Prejean, J., An, Q., Lee, L. M., . . . the HIV Incidence Surveillance Group. (2008). Estimation of HIV incidence in the United States. JAMA: Journal of the American Medical Association, 300, 520–529. 10.1001/jama.300.5.520

Hart, W., Albarracín, D., Eagly, A. H., Brechan, I., Lindberg, M. J., & Merrill, L. (2009). Feeling validated versus being correct: A meta-analysis of selective exposure to information. Psychological Bulletin, 135, 555–588. 10.1037/a0015701

Hennessy, M., Mercier, M. M., Williams, S. P., & Arno, J. N. (2002). Client preferences for STD/HIV prevention programs. Evaluation and Program Planning, 25, 117–124. 10.1016/S0149-7189(02)00004-6

Higa, D. H., Marks, G., Crepaz, N., Liau, A., & Lyles, C. M. (2012). Interventions to improve retention in HIV primary care: A systematic review of U.S. studies. Current HIV/AIDS Reports, 9, 313–325. 10.1007/s11904-012-0136-6

Holtgrave, D. R., Valdiserri, R. O., Gerber, A. R., & Hinman, A. R. (1993). Human immunodeficiency virus counseling, testing, referral, and partner notification services. A cost-benefit analysis. Archives of Internal Medicine, 153, 1225–1230. 10.1001/archinte.1993.00410100057008

Johnson, B., & Eagly, A. H. (1989). The effects of involvement on persuasion: A meta-analysis. Psychological Bulletin, 106, 290–314. 10.1037/0033-2909.106.2.290

Johnson, B. T., Scott-Sheldon, L. A., Smoak, N. D., LaCroix, J. M., Anderson, J. R., & Carey, M. P. (2009). Behavioral interventions for African-Americans to reduce sexual risk of HIV: A meta-analysis of randomized controlled trials. Journal of Acquired Immune Deficiency Syndromes (1999), 51, 492. 10.1097/QAI.0b013e3181a28121

Kamb, M. L., Fishbein, M., Douglas, J. M., Jr., Rhodes, F., Rogers, J., Bolan, G., . . .for the Project RESPECT Study Group. (1998). Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted diseases: A randomized controlled trial. JAMA: Journal of the American Medical Association, 280, 1161–1167. 10.1001/jama.280.13.1161

Katz, I. T., Dietrich, J., Tshabalala, G., Essien, T., Rough, K., Wright, A. A., . . .Ware, N. C. (2015). Understanding treatment refusal among adults presenting for HIV-testing in Soweto, South Africa: A qualitative study. AIDS and Behavior, 19, 704–714. 10.1007/s10461-014-0920-y

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498. 10.1037/0033-2909.108.3.480

Lauby, J., Kotranski, L., Feighan, K., Collier, K., Semaan, S., & Halbert, J. (1996). Effects of intervention attrition and research attrition on the evaluation of an HIV prevention program. Journal of Drug Issues, 26, 663–677. 10.1177/002204269602600310

Lauver, D. R., Ward, S. E., Heidrich, S. M., Keller, M. L., Bowers, B. J., Brennan, P. F., . . .Wells, T. J. (2002). Patient-centered interventions. Research in Nursing & Health, 25, 246–255. 10.1002/nur.10044

Lewin, K. (1926). Intention, will and need. Psychologische Forschung, 7, 330–385. 10.1007/BF02424365

Linnan, L. A., Emmons, K. M., Klar, N., Fava, J. L., LaForge, R. G., & Abrams, D. B. (2002). Challenges to improving the impact of worksite cancer prevention programs: Comparing reach, enrollment, and attrition using active versus passive recruitment strategies. Annals of Behavioral Medicine, 24, 157–166. 10.1207/S15324796ABM2402_13

Liu, J., Jones, C., Wilson, K., Durantini, M. R., Livingood, W., & Albarracín, D. (2014). Motivational barriers to retention of at-risk young adults in HIV-prevention interventions: Perceived pressure and efficacy. AIDS Care, 26, 1242–1248. 10.1080/09540121.2014.896450

Lorimer, K., Kidd, L., Lawrence, M., McPherson, K., Cayless, S., & Cornish, F. (2013). Systematic review of reviews of behavioural HIV prevention interventions among men who have sex with men. AIDS Care, 25, 133–150. 10.1080/09540121.2012.699672

McClelland, D. C. (1951). Personality. New York, NY: William Sloane Associates. 10.1037/10790-000

McMahon, R. C., Malow, R. M., Jennings, T. E., & Gómez, C. J. (2001). Effects of a cognitive-behavioral HIV prevention intervention among HIV negative male substance abusers in VA residential treatment. AIDS Education and Prevention, 13, 91–107. 10.1521/aeap.13.1.91.18921

Meader, N., Semaan, S., Halton, M., Bhatti, H., Chan, M., Llewellyn, A., & Des Jarlais, D. C. (2013). An international systematic review and meta-analysis of multisession psychosocial interventions compared with educational or minimal interventions on the HIV sex risk behaviors of people who use drugs. AIDS and Behavior, 17, 1963–1978. 10.1007/s10461-012-0403-y

Mensch, B. S., Hewett, P. C., & Erulkar, A. S. (2003). The reporting of sensitive behavior by adolescents: A methodological experiment in Kenya. Demography, 40, 247–268. 10.1353/dem.2003.0017

Minder, C. E., Müller, T., Gillmann, G., Beck, J. C., & Stuck, A. E. (2002). Subgroups of refusers in a disability prevention trial in older adults: Baseline and follow-up analysis. American Journal of Public Health, 92, 445–450. 10.2105/AJPH.92.3.445

Molden, D. C., & Higgins, E. T. (2005). Motivated thinking. In K. J.Holyoak & R. G.Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 295–317). Cambridge, MA: Cambridge University Press.

Morgan, S., & Yoder, L. H. (2012). A concept analysis of person-centered care. Journal of Holistic Nursing, 30, 6–15. 10.1177/0898010111412189

Morwitz, V. G., Johnson, E., & Schmittlein, D. (1993). Does measuring intent change behavior?Journal of Consumer Research, 20, 46–61. 10.1086/209332

National Institute of Drug Abuse. (1991). Risk behavior assessment (October ed.). Rockville, MD: NIDA Community Research Branch.

National Institute of Drug Abuse. (1993). Risk behavior assessment (1st ed.). Ann Arbor, MI: Inter-University Consortium for Political and Social Research.

Needle, R., Fisher, D. G., Weatherby, N., Chitwood, D., Brown, B., Cesari, H., . . .Braunstein, M. (1995). Reliability of self-reported HIV risk behaviors of drug users. Psychology of Addictive Behaviors, 9, 242–250. 10.1037/0893-164X.9.4.242

Noguchi, K., Albarracín, D., Durantini, M. R., & Glasman, L. R. (2007). Who participates in which health promotion programs? A meta-analysis of motivations underlying enrollment and retention in HIV-prevention interventions. Psychological Bulletin, 133, 955–975. 10.1037/0033-2909.133.6.955

Nuttin, J. (1980). Motivation, planning and action. Leuven, Belgium: Leuven University Press.

Packel, L., Keller, A., Dow, W. H., de Walque, D., Nathan, R., & Mtenga, S. (2012). Evolving strategies, opportunistic implementation: HIV risk reduction in Tanzania in the context of an incentive-based HIV prevention intervention. PLoS ONE, 7, e44058. 10.1371/journal.pone.0044058

Ponterotto, J. G., Alexander, C. M., & Grieger, I. (1995). A multicultural competency checklist for counseling training programs. Journal of Multicultural Counseling and Development, 23, 11–20. 10.1002/j.2161-1912.1995.tb00262.x

Ponterotto, J. G., Potere, J. C., & Johansen, S. A. (2002). The Quick Discrimination Index: Normative data and user guidelines for counseling researchers. Journal of Multicultural Counseling and Development, 30, 192–207. 10.1002/j.2161-1912.2002.tb00491.x

Prislin, R., & Wood, W. (2005). Social influence: The role of social consensus in attitude and attitude change. In D.Albarracín, B. T.Johnson, & M. P.Zanna (Eds.), Handbook on attitudes and attitude change (pp. 671–706). Hillsdale, NJ: Erlbaum Inc.

Putnam, R. D. (1911). Making democracy work. Princeton, NJ: Princeton University Press.

Rabinowitz, D. C. (2002). Predictors of attrition in a predominantly Caucasian middle-class clinic-based weight loss program. Dissertation Abstracts International: Section B: The Sciences and Engineering, 62, 5976.

Raj, A., Amaro, H., Cranston, K., Martin, B., Cabral, H., Navarro, A., & Conron, K. (2001). Is a general women’s health promotion program as effective as an HIV-intensive prevention program in reducing HIV risk among Hispanic women?Public Health Reports, 116, 599–607. 10.1016/S0033-3549(04)50093-6

Robinson, J. H., Callister, L. C., Berry, J. A., & Dearing, K. A. (2008). Patient-centered care and adherence: Definitions and applications to improve outcomes. Journal of the American Academy of Nurse Practitioners, 20, 600–607. 10.1111/j.1745-7599.2008.00360.x

Roffman, R. A., Picciano, J. F., Bolan, M., & Kalichman, S. C. (1997). Factors associated with attrition from an HIV-prevention program for gay and bisexual males. AIDS and Behavior, 1, 125–135. 10.1023/B:AIBE.0000002973.21222.bc

Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1994). The health belief model and HIV risk behavior change. In R. J.DiClemente & J. L.Peterson (Eds.), Preventing AIDS: Theories and methods of behavioral interventions (pp. 5–24). New York, NY: Springer Science & Business Media. 10.1007/978-1-4899-1193-3_2

Rutledge, S. E., Roffman, R. A., Picciano, J. F., Kalichman, S. C., & Berghuis, J. P. (2002). HIV prevention and attrition: Challenges and opportunities. AIDS and Behavior, 6, 69–82. 10.1023/A:1014580629524

Schilling, R. F., & Sachs, C. (1993). Attrition from an evening alcohol rehabilitation program. The American Journal of Drug and Alcohol Abuse, 19, 239–248. 10.3109/00952999309002683

Schweitzer, C. N. (1997). A study of the relationship between psychosocial factors and attrition, compliance, and gains in Phase II cardiac rehabilitation. Dissertation Abstracts International: Section A: Humanities and Social Sciences, 58, 0392.

Tobias, C., Wood, S., & Drainoni, M. L. (2006). Ryan White Title I survey: Services for HIV-positive substance users. AIDS Patient Care and STDs, 20, 58–67. 10.1089/apc.2006.20.58

Vanable, P. A., Carey, M. P., Brown, J. L., Littlewood, R. A., Bostwick, R., & Blair, D. (2012). What HIV-positive MSM want from sexual risk reduction interventions: Findings from a qualitative study. AIDS and Behavior, 16, 554–563. 10.1007/s10461-011-0047-3

Veach, L. J., Remley, T. P., Jr., Kippers, S. M., & Sorg, J. D. (2000). Retention predictors related to intensive outpatient programs for substance use disorders. The American Journal of Drug and Alcohol Abuse, 26, 417–428. 10.1081/ADA-100100253

Wagenaar, B. H., Christiansen-Lindquist, L., Khosropour, C., Salazar, L. F., Benbow, N., Prachand, N., . . .Sullivan, P. S. (2012). Willingness of US men who have sex with men (MSM) to participate in Couples HIV Voluntary Counseling and Testing (CVCT). PLoS ONE, 7, e42953. 10.1371/journal.pone.0042953

Warner, L., Klausner, J. D., Rietmeijer, C. A., Malotte, C. K., O’Donnell, L., Margolis, A. D., . . . the Safe in the City Study Group. (2008). Effect of a brief video intervention on incident infection among patients attending sexually transmitted disease clinics. PLoS Medicine, 5, e135. 10.1371/journal.pmed.0050135

Watzlawick, P. (1978). The language of change: Elements of therapeutic communication. New York, NY: Norton & Company Inc.

Williams, M. L., Freeman, R. C., Bowen, A. M., Zhao, Z., Elwood, W. N., Gordon, C., . . .Signes, C. A. (2000). A comparison of the reliability of self-reported drug use and sexual behaviors using computer-assisted versus face-to-face interviewing. AIDS Education and Prevention, 12, 199–213.

Wilson, K., & Albarracín, D. (2015). Barriers to accessing HIV-prevention in clinic settings: Higher alcohol use and more sex partners predict decreased exposure to HIV-prevention counseling. Psychology, Health and Medicine, 20, 87–96. 10.1080/13548506.2014.902484

Wilson, K., Durantini, M. R., Albarracín, J., Crause, C., & Albarracín, D. (2013). Reducing cultural and psychological barriers to Latino enrollment in HIV-prevention counseling: Initial data on an enrollment meta-intervention. AIDS Care, 25, 881–887. 10.1080/09540121.2012.729803

Wyer, R. S., & Albarracín, D. (2005). The origins and structure of beliefs and goals. In D.Albarracín, B. T.Johnson, & M. P.Zanna (Eds.), Handbook of attitudes (pp. 273–322). Hillsdale, NJ: Erlbaum.

Yancey, A. K., Ortega, A. N., & Kumanyika, S. K. (2006). Effective recruitment and retention of minority research participants. Annual Review of Public Health, 27, 1–28. 10.1146/annurev.publhealth.27.021405.102113

Submitted: November 3, 2015 Revised: May 9, 2016 Accepted: July 4, 2016

A Method for Making Inferences in Network Analysis: Comment on Forbes, Wright, Markon, and Krueger (2017)

Acknowledgement: Douglas Steinley and Kenneth J. Sher were partially supported by National Institutes of Health Grant R01AA023248-01.

We commend the authors—Forbes, Wright, Markon, and Krueger (2017)—for bringing to light one of the issues that has arisen recently when attempting to make inferences about the structure of so-called psychopathology networks. Specifically, Forbes et al. note that there are emerging regarding our ability to determine the extent that network structures obtained from two (or more) samples are similar or different in key respects. That is, do they replicate? Judging the similarity of various symptom network solutions can be challenging for a number of reasons, not the least of which is the number of parameters modeled including the number of edges (paths) connecting nodes (symptoms) and various measures that are derived from these, perhaps most critically, centrality measures that speak to the relative importance of individual symptoms vis a vis the total network. Depending upon the specific measures of centrality under consideration, such centrality measures could represent a powerful way to identify symptoms that are more fundamental (as opposed to accessory) to a given syndrome, bridge distinct symptom networks, or strongly related to other symptoms.

However, to have confidence in the results of any single network estimation procedure, it is important to be able to demonstrate its replicability, that is how similar are the estimated edge weights and centrality parameters across two different samples. Problems in establishing “reproducibility” is an increasing focus of concern in both psychology (e.g., Pashler & Wagenmakers, 2012) and science more generally (e.g., Baker, 2016). As noted above, the sheer number of parameters estimated make establishing the similarity of two more networks challenging and traditional approaches such as intuitive “eye balling” of the magnitude of various measures of agreement, concordance, or replicability (for instance, the Pearson correlation coefficient, the intraclass correlation)—many of which will not be appropriate—fails to provide a framework for making strong inferential statements regarding replicability of network parameterizations. Moreover, for technical reasons, measures of association based on binary data provide special challenges owing to their sensitivity to base rates (i.e., marginal frequencies) which may differ across samples either by design or sampling effects. Having a methodology for assessing network similarity that adequately addresses this challenge would make it possible for researchers to validly judge reproducibility and replicability.

In this commentary, we describe a simple yet effective method for generating empirical confidence intervals for whatever statistic is desired for inspection and demonstrate its use on a sample data set under the conditions described by Forbes et al. (2017). We then go on to provide some general comments regarding its application.

More specifically, we first review the notion of empirically creating appropriate null distributions for quantities of interest. We then discuss the application of such approaches to traditional one-mode networks where one set of nodes that are similar to each other or the same type (e.g., individuals). Following this, we provide an overview of the structure of two-mode matrices as commonly thought of in terms of network structure or graph theory, where there are two different types of nodes (e.g., symptoms and people). After that, we introduce the proposed hypothesis testing for two-mode binary networks and explicate the method by reanalyzing one of the data sets from Forbes et al. (2017) to obtain confidence intervals under an appropriate null distribution for edges and centrality statistics. This allows us to conduct subsequent statistical tests by determining whether the statistic of interest lies within the null confidence interval. If it does, then we conclude that the observed network statistic is performing the same as it would under a random two-mode network; if it does not, then a process of potential interest is likely driving the observed value for the tested statistic.

In multivariate statistics, it is often difficult to derive the exact sampling distribution for many quantities of interest. A standard practice is to generate random data with appropriate constraints, fit the hypothesized model to these generated data, and then compare the quantity of interest when computed under random data with that in the observed network. If they are very similar, then we cannot conclude that the observed data are any different than random data. Such an approach is often employed in other areas of multivariate statistics (e.g., use of parallel analysis in factor analysis, Horn, 1965).

In parallel analysis, the eigenvalues obtained from the covariance (or correlation) matrix derived from the sample data are compared with the eigenvalues obtained from covariance (or correlation) matrices arising from completely random data. The central idea behind of parallel analysis is that if a set of measured variables arises from a common factor model with f factors, then the f largest eigenvalues as computed from the data set should exceed that of the expected (e.g., average) eigenvalues of random data. One critical factor in making this comparison is how the random data are generated. Specifically, the constraints that are placed on each of the data sets in the comparison population are critical. If the eigen-decomposition is on the correlation matrix, a common constraint is the generated data should have the same sample size (e.g., the same number of observations) and the same number of variables. If the eigen-decomposition is on the covariance matrix, an additional constraint for consideration would be to ensure that the randomly generated variables had the same variances.

Similarly, corresponding procedures for comparing observed fit statistics to fit statistics derived from random data has been implemented in the domain of cluster analysis. Specifically, Steinley (2006, 2007, 2008) and Steinley and Brusco (2011) generated distributions based on random data to test the quality of a cluster solution and to determine whether the correct number of clusters had been chosen, respectively. Using this general approach, Steinley (2004) developed a method for sampling cluster agreement-matrices to approximate the statistic’s sampling distribution. Although used extensively in multivariate statistics, such approaches have also seen use in traditional network analyses—as explained in the following section.

One-Mode Networks

For one-mode networks (e.g., the rows and the columns of the adjacency matrix describe the same entities, such as a friendship network) statistical inference has long been an active area of research and methodological development because the standard inferential techniques relying on independence of observations does not apply. As a solution, it has been argued that the testing network statistics of interest can be obtained by generating a null distribution from a population of networks that have an appropriate structure that the researcher wishes to control. Generally, these networks are generated uniformly (e.g., each network has an equal chance of being selected and included in the population. Anderson, Butts, and Carley (1999) described an approach to conduct hypothesis testing on network level statistics when controlling for the number of observations (e.g., the number of actors or nodes) and the overall density (e.g., the number of links). Additional constraints can be added to the generation process, such as controlling for the number of mutual, asymmetric, and null ties. Furthermore, for one-mode networks, the notion of generating networks with specific properties has flourished with recent developments in exponential random graph modeling (see Hunter & Handcock, 2006).

Two-Mode Matrices

Two-mode data binary data has long been of interest in the psychological sciences (Arabie & Hubert, 1990; Arabie, Hubert, & Schleutermann, 1990; Brusco & Steinley, 2006, 2007a, 2007b, 2009, 2011; Brusco, Shireman, & Steinley, in press; Rosenberg, Van Mechelen, & De Boeck, 1996). In terms of networks and graph theory, two-mode binary data can be thought of as a bipartite graph and represented in a standard data matrix with n rows and p columns (where, often, rows represent the subjects and columns represent the variables). In the network literature, these are often deemed affiliation matrices. Until recently, there has been a dearth of methods for generating binary affiliation matrices with fixed margins, with the ones being used somewhat cumbersome and inefficient and too slow for application to large networks (Admiraal & Handcock, 2008; Chen, Diaconis, Holmes, & Liu, 2005). However, within the last few years, a pair of papers have appeared (Harrison & Miller, 2013; Miller & Harrison, 2013) that provide a computationally efficient method for generating affiliation matrices with fixed margins.

Interestingly, the motivating example for this work arises out of ecology to test cohabitation of species in biogeographical data, where the data are often of the form Species × Habitat. As a bit of background, original work by Diamond (1975) made extensive claims about “species assembly rules” that aimed to explain the interaction between specific species and their chosen habitats—indeed, some of the relationships between species and habitats were quite strong. However, Connor and Simberloff (1979) argued that, if one controlled for the base rates of the different species and the different habitats, then many of these so-called “strong” associations could have arisen by random chance. According to Miller and Harrison (2013), this set off years of argument about the nature of null hypothesis testing and whether to control for the marginal distributions or not. Owing to the fact that the effect disappears when the marginal distributions are controlled for, the current guidelines recommend that the marginal distributions are fixed such that observed effects are not due to the base rates of the rows and columns alone.

If the two-mode matrix is viewed as a contingency table, then fixing both marginal distributions corresponds to Fisher’s exact test, with the following assumptions:

1. Each observation is classified into one and only one category of the row variable and into one and only one category of the column variable.
2. The N observations come from a random sample such that each observation has the same probability of being classified into the ith row and the jth column as any other observation.
3. The null hypothesis is: The event of an observation being in a particular row is independent of that same observation being in a particular column.

This framework will serve to generate the random matrices for our test, as described below.

The goal of this commentary is to present such a test based on generating a set(s) of matrices from prespecified marginal distributions. The desire is to provide a statistical context for evaluating the magnitude some measures of correspondence with respect to the expected parameter estimates generated from random processes with known base rates. Further, much like the ecology literature, we argue that in the psychopathology literature it is imperative to compare observed results to null distributions that have the same prevalences of the symptoms themselves and the same distributions of severity across the individuals. This is a necessary condition to begin making precise, generalizable statements about specific symptoms by guaranteeing that any observed/interpreted relationships between the variables is attributable to their co-occurrence within the same individuals are not an artifact imposed by the marginal distributions.

Holding the within-person marginal distributions constant is necessary because the goal of network analysis in some ways is much more lofty that that of the common cause models, such as factor analysis. In the latter, it is often assumed that all items loading onto the same factor should be (theoretically) exchangeable; however, in network analysis the goal is to discover the explicit connections between individual items—as such, the items themselves take on a much more pronounced role in the modeling process. Specifically, for factor models, knowing that a subject endorsed 3 of 5 items on a disorder is often enough; in fact, that is what much of the diagnostic literature is predicated on. Conversely, for network models, the specific 3 items are much more important because the goal is to make causal (or pseudocausal) connections between specific items. The bar is much higher. Likewise, holding the within-item prevalences constant is extremely important as well. Specifically, it can easily be shown that items that have a higher prevalence of occurring are more likely to have observed edges in the network model. Consequently, it is important to model random items with the same rates of occurrence to ensure that observed network edges are not merely a byproduct of items that have been endorsed more than other items. Without such a reference, researchers will remain in the dark about what their findings indicate with these newer network models. The following two subsections describe how the procedure works, with a more mathematical explanation provided in the Appendix.

Algorithm for Within Network Importance

The ingredients for testing so-called “within” network importance are fairly straightforward. Namely, all that is required is the original data matrix Xn×p with n rows and p columns as well as the estimated network statistic of interest, say θ̂. The goal is then to obtain the sampling distribution for θ under the null distribution that the location of the ones (i.e., the presence of the symptom, say) in the binary matrix X - marginal distributions for both the rows and columns.

Procedurally, the process is fairly simple. First, the user must choose the number of random data matrices to generate. After this is chosen, each random matrix is generated such that it has identical column totals (e.g., each item prevalence is the same in the random matrices as in the observed matrix) and row totals (e.g., the severity profiles of the observations in the random matrices are the same as in the observed matrix). However, in the random generation process, the elements in the random matrices are generated such that the rows and columns are independent of each other conditional on the row totals and column totals (e.g., the marginal distributions) being the same. After all of the random matrices are generated, the statistic of interest (e.g., edge weights, centrality parameters, etc.) is computed on each of the random matrices. Finally, the statistic for the observed data is compared with the reference distribution derived from the random matrices, allowing for the calculation of a percentile score. If this score is between the 2.5 percentile and the 97.5 percentile, we can say that the observed data are providing results that are consistent with what we would expect to see in random data. That is, the p values associated with the estimates do not exceed a nominal level of.05.

Algorithm for Between Importance

The first algorithm discusses determining whether the observed data deviates from the what would be observed via random chance and concerns network elements that are internal to one data set (hence, the moniker of “within”). However, it is also possible to compute the correspondence or stability of network statistics across different data sets, which we will term as “between.” To compute the correspondence between two networks, in additional data set, Yn2×p, is required (note that while the number of variables must be the same, the two data sets may have different numbers of observations). Network structures are then derived from both X and Y, and a measure of correspondence is computed. Then, pairs of random matrices are generated, with one corresponding to X and one corresponding to Y, and the respective marginal distributions are fixed to those of the observed matrices. The same measure of correspondence is computed for each pair for random matrices, allowing for the generation of sampling distribution under random chance to be obtained. The computation of percentile scores than proceeds in the same fashion as described for the within network importance.

Relationship to Bootstrapping

Recently, Epskamp, Borsboom, and Fried (in press) have introduced methods for applying bootstrap techniques to psychological networks. These methods are designed to (a) assess the accuracy of estimated network connections, (b) investigate the stability of centrality indices, and (c) test whether network connections and centrality estimates for different variables differ from each other. The primary difference between the methodology proposed above and bootstrapping approach is that the former is assessing whether the observed effects are different than what we would expect from random chance while the latter assess stability and accuracy of coefficient estimation.

As such, it is possible, and as we see in the examples below, perhaps expected (because of the constraints placed on the network space due to the marginal distributions) that estimated network effects can be both simultaneously stable and not different than random chance. Additionally, a confidence interval around the either the connections between variables or the centrality estimates, as derived from the bootstrap approach, cannot include zero and still overlap with (or even fall within) the confidence interval indicating what we would expect by chance alone. In that instance, the estimate would be stable, but not particularly interesting/informative—and, in such a scenario we would caution against interpreting the estimate because, while the effect is shown to exist, it is not different than what would be expected by chance alone. We summarize the four possible outcomes derived from the outcomes of the two testing procedures in Table 1.

Four Potential Outcomes Between Bootstrap Procedure and Random Chance

In Table 1, we provide the more general possibility of testing hypotheses (or constructing confidence interval) concerned with comparing the effects with a prespecified null value, v0. Likewise, we can construct, from the procedure outlined above, the expected value of the effect under random chance, vrc. Then, the combined information of whether the estimate (v̂) is different than the hypothesized value (v0) and/or the value under random chance (vrc) is useful for determining the interestingness/importance of an effect. Obviously, the most interesting effect will be situations where it is shown that the estimate is both different from the hypothesized value and what is expected under random chance. Conversely, if effects are not different from what is expected under random chance, the information derived from the bootstrap method is uninteresting regardless of the degree of stability, accuracy, or difference from the hypothesized value. Lastly, when the effects are different from chance but not different from the hypothesized value, the result could be potentially interesting. In terms of the network diagram, this would correspond to unobserved links between pairs of variables. As such, each observed link can be interesting (different from zero, different from random chance) or uninteresting (not different from random chance); likewise, each unobserved link could be uninteresting (not different from zero, not different from random chance) or potentially interesting (not different from zero, different from random chance). We use “potentially” interesting because relationships (e.g., presence of links) and conditional independence (e.g., absence of links) are both inferred when the network diagram is visually inspected.

The goal of these analyses is to infer the true underlying causal relationships between psychological symptoms, thus a significant result would be one that informs us of relations above and beyond what their prevalence rates tell us. Given the length constraints of the commentary, we focus on demonstrating the within-network evaluation algorithm for the NCS-R data as fit with the Ising model. Specifically, we will demonstrate the construction of confidence intervals around the estimated edge weights and centrality statistics as estimated using the qgraph package in the R statistical computing environment. To begin with, we need the marginal distributions of the NCS-R data. The frequency of each symptom is provided in Table 2, while the severity frequency (e.g., the number of symptoms endorsed by each subject, ranging from 0–18) is provided in Figure 1. From these data, we can immediately see the issue with only fixing one of the distributions. For instance, if we only fixed the prevalence of each symptom, then they would be distributed equally across the subjects, resulting in the average subject being assigned 3.75 symptoms. This would completely ignore the role of the severity continuum and its distribution in forming the network structure. Likewise, if we fixed only the severity marginals, we would see equally distributed counts over all of the symptoms, resulting in each having the same expected prevalence; consequently, differing prevalence rates alone could potentially define the “network” structure.

Frequency of Symptoms in the NCS-R Data

Figure 1. Frequency distribution of number of symptoms in NCS-R data showing the number of individuals with different numbers of symptoms.

Centrality Statistics

We implemented the algorithm as described above. For this small example, we generated 1000 matrices with marginals fixed to the values indicated in Table 2 and Figure 1. For each parameter of interest, we sorted the 1000 values from highest to lowest and created an empirical 99% confidence interval by taking the 5th and 995th ordered value. While we provide 99% confidence intervals here, more discussion is provided at the end of this section regarding choosing appropriate confidence levels. The confidence intervals for the centrality statistics are displayed in Table 3. From Table 3, we see that there are some variables that have significant centrality statistics when compared with what would be expected from chance. Significant values of centrality statistics are noted with an asterisk. For betweenness, we see that not all nonzero centrality statistics are significant, the most egregious being the betweenness score for “Anxiety about > 1 Event” is 38; however, we find that is not significantly different than would be expected by chance. This indicates that the magnitude of the centrality statistics does not necessarily indicate its relevance. We see similar results for closeness centrality, where 33% of the statistics are significant; unfortunately, without a formal test, it is impossible to determine which are different than chance as all of the estimates are within.017 of each other and magnitude alone does not confer significance. For strength, about two thirds of the centrality statistics are significant.

Confidence Intervals for Centrality Statistics

Finally, it is important to note that sometimes the centrality statistics are significantly less than would be expected by chance. This is true for “Chronic Anxiety” for betweenness and closeness, as well as “Muscle Tensions” for strength. This complicates the evaluation of this type of output because it is unknown whether observed results are significantly greater than chance, significantly less than chance, or no different than chance. Once again, the necessity for a formal test is highlighted.

Edge Weights

The difficulty is exacerbated for evaluating the edgeweights as displayed in Table 4. In Table 4, a “+” in the lower-triangle indicates the estimated edgeweight is significantly greater than chance, a “−” indicates the estimated edgeweight is less than chance, and a blank indicates that the value fell within the 99% confidence interval. One of the first things to notice is the prevalence of minus signs in Table 4. This indicates the absence of a link is just as important as the presence of a link. Of the 153 possible links, only 99 were significantly different than chance (65%), indicating that approximately one third of the estimated network is functioning equivalently to what we would expect by chance. Furthermore, with the three possible classifications of any individual edgeweight being (a) significantly greater than expected by chance, (b) significantly less than expected by chance, and (c) no different than expected by chance, the edgeweights are classified into those categories at the almost equal rates of 30%, 35%, and 35%, respectively. This uniform distribution of edgeweights to the three possible outcomes indicates that evaluating the results from these models by mere inspection of the edgeweights themselves is noninformative.

Estimates and Significance for Edge Weights

Once again, however, the specific edgeweights that are significant cannot be deduced by any pattern of edgeweights or their raw magnitude. For instance, the probability of an edgeweight being no different than chance if the observed edgeweight is greater than zero is 44%—so an observed edgeweight greater than zero is almost as equally as likely to be due to chance as not. To make it more salient, that would mean that nearly half of the edges depicted on the left side of Figure 3 in the Forbes et al. article are due to chance alone. Additionally, some of the zeros are included in the confidence intervals and some are not. Specifically, the probability of an edgeweight being significantly worse than chance given the observed edgeweight is less than or equal to zero is 28%. In terms of the graphical representation, that would mean that nearly 3 of 10 of the lines that are absent on the left hand side of Figure 3 are absent because of chance. Such variability in the confidence of the fidelity of observed edges in the figure renders these types of depictions of networks almost useless. This calls for an implementation of a formal testing procedure to assess the relevance of reported edgeweights or their absence.

Figure 2 illustrates the four possible outcomes as described in Table 1. The upper-left panel indicates an “interesting” finding between depressed mood and loss of interest. The edgeweight (1.93) falls above the distribution for random chance, and the bootstrap confidence interval around the edgeweight does not include zero. The lower-left panel indicates a “potentially interesting” connection between sleep problems (depression) and anxiety about more than one event. Specifically, the estimated value of zero is nonsignificant by both the eLasso and the bootstrap confidence interval around the estimate (0.00); however, the distribution based on random chance alone would expect this connection to be greater than what was observed. The final two panels on the right indicate two different “uninteresting” scenarios. The lower right panel shows the distribution for the edgeweight between chronic anxiety and sleep problems from anxiety—both distributions encompass the observed estimate of zero. The upper-right panel indicates a situation, in this case the relationship between loss of interest and psychomotor disturbances. The bootstrap test indicates that the edgeweight is significantly different than zero; however, the distribution derived from random matrices indicates that the observed value of 0.52 is not unexpected.

Figure 2. Examples of bootstrap compared with random distribution.

Confidence Levels

Above, we used a 99% confidence level to help control for multiple tests and illustrate the general notion of the differences between generating confidence intervals that reflect accuracy and stability through a bootstrap procedure versus the types of confidence intervals that aid in determining whether effects are different than would be expected by random chance alone. However, it is imperative to realize that even a more stringent confidence interval level will not aid in overcoming the threat of Type I error rates in the context of so many tests. Generally, there are different tests for edgeweights and 3p tests for centrality measures (e.g., one for each variable on strength, closeness, and betweenness). In Table 4, there are unique effects—for an α = .05, there would be (.05)153 = 7.65 significant effects based on chance alone, with the more stringent α = .01, (.01)(153) = 1.53 significant effects are expected to be attributable to chance alone.

In situations that rely on multiple testing, it is common to correct each individual test’s alpha to obtain an overall familywise α that corresponds to the prespecified value. The most conservative adjustment is the Bonferroni correction, which divides the familywise α by the total number of tests to obtain an appropriate level of α for each individual test—in this case, the adjusted α is (.05)/153 = .0003. As Epskamp, Borsboom, and Fried (in press) indicate, this type of correction can be overly burdensome and lead to the possibility of not finding any of the true effects (e.g., Type II errors). Beyond the inherent inverse relationship between Type I and Type II error, the computational burden to generate the requisite number of bootstrap samples for such small values of α can result in the procedure not being used in practice. For instance, generating enough samples to be able to obtain a 99.97% confidence interval would require around 10,000 samples; however, somewhere in the neighborhood of 50,000–10,000 would be preferable. As such, in their recommendations, Epskamp et al. (in press) indicate that corrections for chance will be relegated to future research, and in the interim, it is appropriate to proceed with uncorrected tests for the edgeweights.

While the sentiment to proceed with testing each test at α = .05 (the default setting in the Espkamp et al. bootstrapping software) is understandable, cautions, nonetheless, are called for. First, the probability of committing at least one Type I error when conducting multiple tests is 1 - (1 - α)^T, where T is the number of tests being conducted. For instance, if you conduct two tests and do not correct for α the probability of committing at least one Type I error is 1 − (.95)² = .0975; however, for the example above, the probability of making at least one Type I error is 1 − (.95)¹⁵³ = .9997—an almost certainty. Furthermore, it is well known that the magnitude of p values cannot be compared to determine which effects are more likely to be significant and which are more likely to be either insignificant or a potentially a Type I error. The inability to do such a ranking is readily seen if one considers alternative correction methods beyond the Bonferroni correction. Specifically, the Holm-Bonferroni method (Holm, 1979) is uniformly more powerful than the Bonferroni correction; however, it proceeds by comparing the lowest p value to the strictest criterion, and then, as p values increase, the comparison criteria becomes less stringent. A similar test that is more powerful, but require assumptions regarding the joint distribution of the test statistics is Hochberg’s step-up procedure (Hochberg, 1988); likewise, one could consider Hommel’s (1988) stagewise rejective multiple test procedure. Unfortunately, each of these procedures would require in increase in the number of bootstrap samples—which would lead to the potential relaxation of the requirement to control for the familywise error rate.

Beyond the potential to incorrectly assume that effects are significant when they are not, the inattention to appropriately control for multiple testing has the potential to call into question all prior research in an area. This can be seen best with the application of multiple testing in fMRI research, another area that relies on a large number of tests and constructs substantive theory from patterns of significance. Recently, Eklund, Nichols, and Knutsson (2016) demonstrated that somewhere in the neighborhood of 40,000 fMRI studies may be invalid due to the inability to control for Type I error rates appropriately. Furthermore, in their conclusion the authors make a particularly relevant statement: “Although we note that metaanalysis can play an important role in teasing apart false-positive findings from consistent results, that does not mitigate the need for accurate inferential tools that give valid results for each and every study.” Because we are in the early stages of developing psychological networks, the most prudent course of action is also establishing appropriate inferential tools immediately and prevent the need to potentially correct hundreds or even thousands of published studies 5, 10, or 20 years from now. This view of tightening the reins for publication is further supported by the recommendation of a group of statisticians who recently weighed in on the reproducibility of psychological science (Johnson et al., 2017): “The results of this reanalysis provide a compelling argument for both increasing the threshold required for declaring scientific discoveries and for adopting statistical summaries of evidence that account for the high proportion of tested hypotheses that are false.” In fact, they suspect that potentially 90% (!!!) of tests performed in psychology experiments are testing negligible effects.

Fidelity of Data

Lastly, we note that there are several decisions made at the data analytic level that can affect and alter the results of any given network model. Some examples would include how the data are collected or how the responses are coded. For instance, it is possible that specific criteria could be coded differently depending on the interview employed to collect the data. Another example would be the so-called “skip out” issue that can occur in epidemiological questionnaires. For “skip out” items, there are gateway items that are assessed first—if the response to the gateway item is negative, then the remainder of the items are not asked, whereas if the response to the gateway item is affirmative, then all subsequent related items are asked. A question arises on how to handle the subsequent items if they were not assessed. A common approach is to assume that if the gateway item is not endorsed, then all subsequent subitems are not endorsed either—this is akin to a logical data imputation. Whether this is appropriate or not will depend on the specific set of questions and criteria being assessed. If it is inappropriate, the estimated effects will be biased; however, that being said, these biases will not effect the validity of the testing approach described above, nor does it mitigate the issues with not controlling for multiple testing.

Replicability

From these findings, we see that the results of many network analyses, as conducted on binary data, might be overstating their findings. Although space limitations only allowed for a modest analysis of the performance the Isingfit model on one data set, current work is expanding the investigation to a broader class of binary matrices as well as extending to the three other types of network models mentioned by Forbes et al. (2017). Preliminary results indicate that the findings will be similar to what was observed for the Isingfit model. In short, we second the conclusion of Forbes et al. and offer the addendum that discovering believable results, at the specific symptom level (whether that is the relationship between pairs of symptoms) or variable level statistics (e.g., centrality statistics), will likely be much more difficult than previously envisioned. As shown above, this is because empirical findings are difficult to distinguish from random chance, and we do not believe that it would be too strong of a suggestion that previously published findings using this methodology should be reevaluated using the above testing procedure. Without this additional testing, future research based on existing findings will likely lead to a significant degree of nonreplicability as the findings are potentially no difference than chance. While the network approach represents an important alternative view of diagnostic systems that could provide new insights into both the basic structure of psychopathology and identify promising targets for intervention by virtue of their centrality, existing methods must be considered unproven. Research practitioners must appreciate the limitations of the existing state-of-the-art and develop and refine approaches likely to provide more robust and interpretable solutions.

Are Psychopathology “Networks” Actually Networks?

In the first footnote, we note that much of the terminology in recent psychopathology network analysis has been borrowed from the traditional network analysis literature—much of which is rooted in psychology and sociology (see Wasserman & Faust, 1994). To determine whether the transferability of methods in traditional network analysis to psychopathology networks is warranted (or should be taken at face value), it is worth highlighting the differences between the two. Generally, there are two types of networks that can be considered: (1) networks that directly assess the relationships between the same set of observations (e.g., one-mode matrices as described above), and (2) affiliation networks where the connections are assessed between two sets of observations (two-mode matrices as described above). Clearly, psychopathology networks fall into the class of affiliation matrices where the connections are measured between observation and diagnostic criteria. The relationships between the criteria are then then derived by transforming the two-mode affiliation matrix to a one-mode so-called “overlap/similarity” matrix between the criteria, where traditional network methods are applied to this overlap/similarity matrix. Faust (1997) indicated that potential pitfalls arise when applying standard centrality measures to networks derived from affiliation matrices: “In going from the affiliation relation to either the actor co-membership relation or the event overlap relation, one loses information about the patterns of affiliation between actors and events. Thus, one needs to be cautious when interpreting centralities for these one-mode relations” (p. 189). The concerns of Faust (1997) are related directly to the concerns raised in the introduction where the methods developed for one-mode networks are applied to the two-mode networks (e.g., affiliation network or bipartite graph).

As Johnson et al. (2017) argue, the editorial policies (and funding priorities) must be compelled to adapt to higher standards prior to putting their stamp of approval on results. Given the nature of problems that confront psychometric network modeling, including (but not limited to): (a) the possibility of observing “significant” effects that are not different than random chance, (b) the difficulties induced by conducting numerous significant tests, which is not controlled for via bootstrapping, and (c) the uncertainty regarding applying traditional methodology developed for one-mode networks to two-mode networks, we wholeheartedly agree with the recommendation by Forbes and company to turn a skeptical eye toward these models. Additionally, given the results from the example above and the theoretical issues that surround multiple testing and appropriate reference distributions, we do not agree with Epskamp, Borsboom, and Fried (in press) that it is reasonable to continue conducting tests at the α = .05 while we wait for methodologists to develop procedures that address the shortcomings. Rather, it is the other way around: wait until methodologists develop the appropriate fixes then proceed with fitting these models. Not following such an approach runs the very real risk of creating a series of publications that contain results that are not reproducible and likely no different than what is expected under one of the most basic models of chance in all of categorical data analysis.

Although we have provided numerous caveats and cautions to employing psychometric network models, we do believe that they are opening a potentially important area of research for us to consider. Specifically, the moving more to a causal structure model, as opposed to a common cause (such as latent variable) model holds great appeal as the parallelism to traditional medical models of diseases is enhanced. In our view, it is likely that the true model (or at least the best fitting models) will be a hybrid of network models and latent variable models. Further, although we provide extreme caution in using these methods to motivate theory, we do believe that such techniques can be a useful tool in the arsenal of the researcher when used to supplement a rich, substantive knowledge regarding the psychopathology being studied. Additionally, we note that by using correlations (or functions thereof) the network models have wed themselves closely to traditional latent variable models in terms of how the relationships between variables are conceptualized. However, there is a rich history in assessing the similarity between binary items in general and elements within a bipartite graph specifically. Broadening approaches beyond the logistic regression framework encapsulated by the Isingfit approach (and perhaps even abandoning the notion of correlation as the foundational measure of association) could open the entire field of psychometric networks to entirely new horizons. Finally, many of the concerns raised in this commentary could be alleviated if we move from the current state of exploratory psychometric networks to confirmatory psychometric networks, allowing a priori specified effects to be tested rather being hamstrung out of the gate by the need to correct for multiple testing.

¹ We note that there is a good deal of appropriation of terminology from the social network analysis literature used in these analyses; we would caution too much transference as the psychopathology networks are induced from bipartite graphs versus being observed directly.

² The eigenvalues from the completely random data are obtained by averaging over several—usually one thousand or more—sets of eigenvalues that have been obtained from independently generated random data.

³ Such an approach has been superceded by the derivation of a closed-form solution for comparing the equivalence of two partitions (Steinley, Brusco, & Hubert, 2016), but the basic logic still applies.

⁴ We note that most of the time v0 = 0.

⁵ Note that under many situations, the expected value of the effect is often zero (e.g., no effect). For instance, in the simple case of a t test or a linear regression model, generating data from random distributions—accomplished by randomly assigning group labels for the t test or randomly generating predictors uncorrelated with the dependent variable for regression—we would expect mean differences of zero and slopes of zero. The deviation from no effect for these psychological networks is due to the restriction of the range of the sample space of potential networks that is induced from the observed marginal distributions.

⁶ The between network importance algorithm would be implemented in a similar fashion, but because of space constraints we have not included an example here.

⁷ The data set was provided to us by the first author, Miriam Forbes, of the target article.

Admiraal, R., & Handcock, M. S. (2008). networksis: A package to simulate bipartite graphs with fixed marginals through sequential sampling. Journal of Statistical Software, 24, 1–21.

Anderson, B. S., Butts, C., & Carley, K. (1999). The interaction of size and density with graph-level indices. Social Networks, 21, 239–267.

Arabie, P., & Hubert, L. J. (1990). The bond energy algorithm revisited. IEEE Transactions on Systems, Man, and Cybernetics, 20, 268–274.

Arabie, P., Hubert, L., & Schleutermann, S. (1990). Blockmodels from the bond energy algorithm. Social Networks, 12, 99–126.

Baker, M. (2016). Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the crisis rocking science and what they think will help. Nature, 533, 452–455.

Brusco, M. J., Shireman, E., & Steinley, D. (in press). A comparison of latent class, K-means, and K-median method for clustering dichotomous data. Psychological Methods.

Brusco, M. J., & Steinley, D. (2006). Inducing a blockmodel structure on two-mode data using seriation procedures. Journal of Mathematical Psychology, 50, 468–477.

Brusco, M. J., & Steinley, D. (2007a). Exact and approximate algorithms for part-machine clustering based on a relationship between interval graphs and Robinson matrices. IIE Transactions, 39, 925–935.

Brusco, M. J., & Steinley, D. (2007b). An evaluation of a variable-neighborhood search method for blockmodeling of two-mode binary matrices based on structural equivalence. Journal of Mathematical Psychology, 51, 325–338.

Brusco, M. J., & Steinley, D. (2009). Integer programs for one- and two-mode blockmodeling based on prespecified image matrices for structural and regular equivalence. Journal of Mathematical Psychology, 53, 577–585.

Brusco, M. J., & Steinley, D. (2011). A tabu search heuristic for deterministic two-mode blockmodeling of binary network matrices. Psychometrika, 76, 612–633.

Chen, Y., Diaconis, P., Holmes, S. P., & Liu, J. S. (2005). Sequential Monte Carlo methods for statistical analysis of tables. Journal of the American Statistical Association, 100, 109–120.

Connor, E. F., & Simberloff, D. (1979). The assembly of species communities: Chance or competition?Ecology, 60, 1132–1140.

Diamond, J. M. (1975). Assembly of species communities. In M.Cody & J.Diamond (Eds.), Ecology and evolution of communities (pp. 342–444). Cambridge, MA: Harvard University Press.

Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the National Academies of Science of the United States of America, 113, 7900–7905.

Epskamp, S., Borsboom, D., & Fried, E. I. (in press). Estimating psychological networks and their accuracy: A tutorial paper. Behavior Research Methods.

Faust, K. (1997). Centrality in affiliation networks. Social Networks, 19, 157–191.

Forbes, M. K., Wright, A. G. C., Markon, K. E., & Krueger, R. F. (2017). Evidence that psychopathology symptom networks have limited replicability. Journal of Abnormal Psychology, 126, 1011–1016.

Harrison, M. T., & Miller, J. W. (2013). Importance sampling for weighted binary random matrices with specified margins. arXiv:1301.3928 [stat.CO]

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800–802.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75, 383–386.

Horn, J. L. (1965). A rationale and technique for estimating the number of factors in factor analysis. Psychometrika, 30, 179–185.

Hunter, D. R., & Handcock, M. S. (2006). Inference in curved exponential family models for networks. Journal of Computational and Graphical Statistics, 3, 565–583.

Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2017). On the reproducibility of psychological science. Journal of the American Statistical Association, 112, 1–10.

Miller, J. W., & Harrison, M. T. (2013). Exact sampling and counting for fixed-margin matrices. The Annals of Statistics, 41, 1569–1592.

Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on the replicability in psychological science: A crisis of confidence?Perspectives on Psychological Science, 7, 528–530.

Rosenberg, S., Van Mechelen, I., & De Boeck, P. (1996). A hierarchical classes model: Theory and method with applications in psychology and psychopathology. In P.Arabie, L. J.Hubert, & G.De Soete (Eds.), Clustering and classification (pp. 123–155). River Edge, NJ: World Scientific.

Steinley, D. (2004). Properties of the Hubert-Arabie adjusted Rand index. Psychological Methods, 9, 386–396.

Steinley, D. (2006). Profiling local optima in K-means clustering: Developing a diagnostic technique. Psychological Methods, 11, 178–192.

Steinley, D. (2007). Validating clusters with the lower bound for sum of squares error. Psychometrika, 72, 93–106.

Steinley, D. (2008). Stability analysis in K-means clustering. British Journal of Mathematical and Statistical Psychology, 61, 255–273.

Steinley, D., & Brusco, M. J. (2011). Testing for validity and choosing the number of clusters in K-means clustering. Psychological Methods, 16, 285–297.

Steinley, D., Brusco, M. J., Hubert, L. (2016). The variance of the adjusted Rand index. Psychological Methods, 21, 261–272.

Wasserman, S. W., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge, UK: Cambridge University Press.

APPENDIX A: Algorithms for Testing Replicability when Marginal Distributions are Fixe

The ingredients for testing so-called “within” replicability are fairly straightforward. Namely, all that is required is the original data matrix Xn×p with n rows and p columns as well as the network statistic of interest, say θ̂. The goal is then to obtain the sampling distribution for θ under the null distribution that the location of the ones in the binary matrix X are random subject to the observed marginal distributions for both the rows and columns.

The algorithm proceeds as follows:

1. Compute θ̂ from X; compute r_x and c_x (the marginal distributions for the rows and columns, respectively, which in this case are just sums across the columns and rows).
2. Choose the number of random data matrices, R, to generate.
3. Generate the i^th random matrix, M_i, such that r_mi = r_x and c_Mi = c_x (e.g., the marginal distributions of the random matrices will be equivalent to those of the observed matrix, X).
4. Compute θ_i for i = 1, . . ., R. Order the θ’s from smallest to largest, letting the ordered vector of θ be denoted as θ^(o).
5. Create a confidence interval under the null distribution of random association as .
6. If the observed test statistic, θ̂, falls within the interval in Step 5 then we cannot reject the null. Consequently, we conclude the observed value could have arisen by random chance alone.

Algorithm for Between Replicability

The algorithm for testing so-called “between” replicability is almost identical. All that is required is a second data matrix, Yn2×p, with which we want to compare the original data matrix xn1×p. The process proceeds as follows:

1. Compute θ̂_XY from f(X, Y), where f(·) is a function of correspondence between the two structures uncovered from X and Y (for instance, the function could be the correlation of the edge weights between the two network structures). Additionally, compute r_x and c_x from X; compute from Y, r_Y and c_Y from Y.
2. Choose the number of pairs of random data matrices, R, to generate.
3. Generate the i^th pair of random matrices (M_i, N_i), such that r_Mi = r_x and c_Mi = c_x and r_Ni = r_Y and c_Ni = c_Y.
4. Compute θ_i for i = 1, . . ., R. Order the θ’s from smallest to largest, letting the ordered vector of θ be denoted as θ^(o).
5. Create a confidence interval under the null distribution of random association as .
6. If the observed test statistic, θ̂, falls within the interval in Step 5 then we cannot reject the null. Consequently, we conclude the observed value could have arisen by random chance alone.

Submitted: May 17, 2017 Revised: July 19, 2017 Accepted: July 20, 2017

An Empirical Examination of Distributional Assumptions Underlying the Relationship Between Personality Disorder Symptoms and Personality Traits

Acknowledgement: This research was supported by grants (MH45448, Lenzenweger; F31MH087053, Wright) from the National Institute of Mental Health, Washington, DC. The views and opinions contained within are solely those of the authors and do not reflect the official position of the funding source.

This manuscript reports on work that represents a portion of the first author's dissertation. We thank Jerry S. Wiggins for providing consultation on the initial use of the Revised Interpersonal Adjectives Scales–Big Five, and Lauren Korfine for project coordination in the early phase of the study. We are indebted to Michael N. Hallquist for generously sharing his time to instruct the first author on the intricacies of the R statistical package. Emily B. Ansell, Nicholas R. Eaton, Christopher J. Hopwood, and Kristian E. Markon each provided helpful comments on an early draft of this work.

Personality disorder (PD) researchers have called for an integration of normal and pathological personality functioning within comprehensive dimensional models of personality (e.g., Depue & Lenzenweger, 2005; Widiger, Livesley, & Clark, 2009). It has been argued that normal personality exists on a continuum of functioning with PDs (Widiger & Trull, 2007). A large empirical literature examining the relationship between traits and PDs contributed to the seminal decision to use a dimensional trait system to conceptualize phenotypic variation in PD in DSM-5 (Skodol et al., 2011). This research relies primarily on cross-sectional correlations and linear regression to model the relationship between personality traits and PD symptoms. However, these analytic tools suffer from limitations when the underlying distribution of the variables is severely non-normal, as is the case with PD symptoms in the population. Alternative approaches that more accurately model the observed PD symptom distribution may provide better estimates of the relationship between personality and its disorder, and may offer new insights into the nature of that relationship.

The extant nosology of PDs represents personality pathology as a collection of “distinct clinical syndromes” (p. 689; American Psychiatric Association, 2000) that differ categorically from normative functioning and each other. These distinctions have been criticized as arbitrary and lacking in robust scientific support (Widiger & Trull, 2007), and diagnostic criteria treated as dimensional markers for disorders perform better by empirical standards (Morey et al., 2007). However, measuring disorders dimensionally cannot confirm their continuity with normal functioning (Lenzenweger & Clarkin, 2005), and the accumulated research presents a mixed picture related to this issue. Meta-analyses (Samuel & Widiger, 2008; Saulsman & Page, 2004) demonstrate that basic personality traits exhibit significant and replicable relationships to PD, but the association between traits and PDs are generally only modest in size (Mdn |r| = .15; range = .02–.54; Samuel & Widiger, 2008). Furthermore, in regression models, the five-factor domains and facets generally only explain a minority of the variance in PDs (e.g., Bagby, Costa, Widiger, Ryder, & Marshall, 2005). Thus, normal personality traits and PD are not interchangeable representations of functioning, despite clearly recognizable shared content (Krueger et al., 2011). Other researchers and theorists have highlighted the general impairment associated with PDs (Kernberg, 1984; Livesley & Jang, 2005; Pincus, 2005). Relatedly, a number of investigators have found that PD is primarily characterized by higher neuroticism, lower conscientiousness, and lower agreeableness, with less in the way of distinction beyond this core trait profile (Hopwood et al., 2011; Morey et al., 2002; Saulsman & Page, 2004). Thus, it may be that a particular combination of traits reflects personality pathology generally, with further differentiation occurring in the presence of this profile.

The question of continuity in personality and its pathology has taken center stage in the development of the DSM-5, which will shift to a dimensional model based on the robust empirical findings suggesting that PD is fundamentally dimensional in nature (Skodol et al., 2011). However, in part because of issues raised here, DSM-5 will distinguish between personality dysfunction and the description of that dysfunction using pathological traits (Hopwood et al., 2011; Krueger et al., 2011). Indeed, the sum of the empirical literature leaves an unclear picture of how PD and personality traits are related to each other. It may be that PD is dimensionally continuous with basic personality traits (e.g., Depue & Lenzenweger, 2005). Alternatively, continuities and discontinuities may exist in these relationships, with some driven by the presence of PD and others, perhaps more subtle, driven by the severity of PD beyond its presence (see Lenzenweger & Clarkin, 2005). What is clear is that the theoretical goal of integrating normative personality traits and PD remains elusive.

The key theoretical questions of how personality and PD relate to one another are also inherently questions of methodology. Dimensional approaches can make varied distributional assumptions that may have relevance for advancing the understanding of the relationship between normal and abnormal personality. Research examining this question relies almost exclusively on standard correlation and linear regression. These approaches make several important assumptions (i.e., normality of residuals, homoscedasticity, linearity of relationship, independence), that, when violated, can bias estimation (Cohen, Cohen, West, & Aiken, 2003). Less serious are biased standard errors, which can produce incorrect significance tests for parameters. More serious violations occur when the actual effect of a relationship is misestimated. A major contributing source to the violation of these assumptions is the distribution of the variables being modeled.

In the population, the actual distribution of PD symptoms is highly positively skewed with a majority of individuals suffering from no symptoms. Figure 1 provides an example of such a distribution using the narcissistic personality disorder (NPD) features in the first wave of the Longitudinal Study of Personality Disorders (LSPD; Lenzenweger, 1999), the dataset used here. This histogram is characteristic of a count distribution. Modeling techniques for counts are primarily based on the Poisson and Negative-Binomial (NB) distributions and can be used for regression when appropriate (Cameron & Trivedi, 1998; Long, 1997). Of note is the large number of zeros in the distribution, which has important implications for modeling the relationship between the symptoms and other variables of interest. These zeros carry important information about who does and does not possess symptoms of PD. With large numbers of zeros in the data, two potential modifications to basic count models are recommended: zero-inflated and hurdle models (Atkins & Gallop, 2007; Cameron & Trivedi, 1998; Long, 1997). Zero-inflated models estimate a group of individuals based on the excess of zeros beyond a standard Poisson or NB model, which are treated as individuals who can only take on a zero value. Hurdle models make a binary distinction between those who have a zero value versus those who have a nonzero value. Despite this distinction in the treatment of zero-values, both models estimate separate regression coefficients for the zero versus nonzero (e.g., absence vs. presence of PD) and the count (e.g., severity of PD) portion of the models. These models are ideal for evaluating whether the traits that give rise to any symptoms of PDs are the same as those that predict the number of symptoms once present.

Figure 1. Observed Longitudinal Study of Personality Disorders (LSPD) narcissistic personality disorder features.

The current study explores the relationship of personality traits to PD symptoms using regression models capable of appropriately modeling the distribution of symptoms in the population. We use the LSPD sample, which is made up of participants recruited both with and without significant pathology, unlike samples selected based on shared diagnostic status or for high levels of pathology. As a result, the distributions of PD symptoms closely matches those found in epidemiological samples (Lenzenweger, 2008). The LSPD dataset is ideal for the types of investigations pursued here because it captures the boundary between those individuals whose personalities function well and those who evidence dysfunction.

Our first aim was to evaluate whether the assumptions of linear regression are violated when predicting PD symptoms and to compare Poisson, zero-inflated Poisson (ZIP), Poisson hurdle (PH), negative-binomial (NB), zero-inflated negative-binomial (ZINB), and negative-binomial hurdle (NBH) regression models that predict PD symptoms from personality trait scores. Our second and more substantive aim involves comparing the patterns of significant regression coefficients to determine the effect of varying distributional assumptions on the relationship between traits and PDs. Attention is also given to differences in patterns in the prediction of presence versus severity of PD features in the zero-inflated/hurdle models.

Participants

Detail concerning the participant selection procedure in the LSPD is given elsewhere (Lenzenweger, 1999). The 250 participants were drawn from a nonclinical university population, are balanced on gender (53% women), and the mean age at entry into the study was 18.88 years (SD = 0.51). Approximately half of the participants were selected based on putative positive PD status as assessed by a self-report PD screener. This ensured an adequate sampling of PD pathology in a nonclinical population. Based on diagnostic assessments conducted by experienced clinicians, 11% of the participants qualified for a PD diagnosis of some sort, and 45.2% met the criteria for an Axis I diagnosis (Lenzenweger, 1999). Rates of diagnosed PDs in the sample were as follows: paranoid = 1.2%, schizoid = 1.2%, schizotypal = 1.6%, antisocial = 0.8%, borderline = 1.6%, histrionic = 3.5%, narcissistic = 3.1%, obsessive–compulsive = 1.6%, passive-aggressive = 0.8%, avoidant = 1.2%, dependent = 0.8%, and not otherwise specified = 4.3%. It is important to note that these rates closely mirror the rates of PD found in large epidemiological samples (Lenzenweger, 2008).

Measures

At baseline, participants completed a diagnostic clinical interview and self-report personality measures. Only data from these initial assessments are used here.

The International Personality Disorder Examination (IPDE; Loranger, 1999) was used to assess PD symptomatology. The DSM–III–R criteria were assessed in this study because these were in effect at the time the LSPD was undertaken. The interrater reliability for IPDE assessments (based on intraclass correlation coefficients) was excellent, ranging between .84 and .92 for all PD dimensions used for this study. For each symptom, an individual may receive a score of 0 (absent or normal), 1 (exaggerated/accentuated), or 2 (criterion/pathological). These values are summed within each disorder to create a count of disorder related features.

The Revised Interpersonal Adjective Scales–Big Five (IASR-B5; Trapnell & Wiggins, 1990) contains 124 trait descriptive adjectives rated on a 0 to 8 scale that provides scores for the personality trait dimensions of Dominance, Affiliation, Conscientiousness, Neuroticism, and Openness. Coefficient alphas ranged from .82 to .96.

A series of regression models were estimated in R Package PSCL (Zeileis, Kleiber, & Jackman, 2008). Each PD's count and the Total PD count were regressed on each personality trait score separately. A model was estimated for each personality trait separately in keeping with past literature, and because the traits are orthogonal in theory, but in practice often exhibit relationships that attenuate regression coefficients when entered simultaneously in a model. Trait predictors were standardized. For each trait-PD pairing, a set of models was estimated with Normal, Poisson, ZIP, PH, NB, ZINB, and NBH distributions specified for the PD counts.

The linear regression models were evaluated by testing linearity, normality of the residuals, and homoscedasticity. A minority (22%) of the models violated the assumption of linearity. However, all model residuals exhibited significant skewness (M = 2.75; range = 1.83–4.41) and excess kurtosis (M = 10.50; range = 3.59–25.89). Visual analyses suggested that the assumption of homoscedasticity was untenable for all models. In addition, 36% of the models predicted negative symptom counts. Thus, the assumptions of linear regression models do not hold and a considerable proportion result in the prediction of impossible values, all of which emphasizes the need for count based models.

Poisson regression models make strict assumptions about the variance of the observations that are frequently violated in applied research (Coxe, West, & Aiken, 2009). The NB differs from the Poisson in that it estimates an additional parameter for the variance of counts, and therefore these two models are nested and can be compared using likelihood ratio tests (LTRs; Long, 1997). In each case, the NB model was a significant improvement over its Poisson counterpart. Models which account for the large stack of zeros in the data using either zero-inflation or a hurdle are non-nested relative to the basic count models and each other, and therefore require an alternative test. To compare non-nested models, we employed a Vuong (1989) non-nested LRT. The Vuong LRT compares two models under the null hypothesis “that the competing models are equally close to the true data generating process” (p. 307) such that a significant statistic favors one model over the other, and a nonsignificant statistic suggests that the models are equivalent. When models are quantitatively equivalent additional criteria are required to select between models. One common approach is to select the model with the fewest estimated parameters, placing a premium on parsimony. Yet there may be a conceptual/theoretical preference for considering more complex models under these conditions. As Atkins and Gallop (2007) argue, a histogram like that depicted in Figure 1 would appear to suggest two separate processes: There are those individuals without any personality pathology, and among those that have it, a range of severity. They go on to note that the two-step count models are well-suited for investigating “psychological models in which there are two processes and where the determinants of those processes differ” (p. 733). Our results of a comprehensive comparison of models suggest that with few exceptions, the NB, ZINB, and NBH models are equivalent. When a model was favored, it was a two-step model, and between those the hurdle models. In addition, a number of ZINB models were nonidentified. Therefore, we retain for consideration the NB and NBH models.

Table 1 reports the regression coefficients for the Normal, NB, and NBH and their significance. No formula exists for transforming the different results to the same effect size for direct comparison across models. Therefore, what is most informative is the valence and significance level of each coefficient. Odds/rate ratios of 1.0 indicate no effect, and those below 1.0 indicate a negative association between predictor and outcome. Given the number of coefficients we highlight notable findings, and refer readers to Table 1 for a more detail. Importantly, similar patterns emerge across all models. Specifically, radical differences, such as a valence change, do not occur. However, pattern differences in significant coefficients across steps of the NBH models suggest different processes associated with the presence versus severity of PD symptoms of different types. We briefly summarize these findings organized around the trait dimensions.

Summary of Coefficients From Models Regressing Personality Disorder Symptoms on Personality TraitsSummary of Coefficients From Models Regressing Personality Disorder Symptoms on Personality Traits

Dominance most often predicted PD presence (schizoid, histrionic, dependent, obsessive-compulsive) and less commonly PD severity (narcissistic) or both (schizotypal, avoidant). Affiliation, in contrast, tended to predict both presence and severity (schizoid, schizotypal, antisocial, borderline, narcissistic, and obsessive-compulsive) and otherwise just presence (paranoid, avoidant) when considering individual PDs. Conscientiousness only predicted PD presence (paranoid, borderline, narcissistic, dependent). Neuroticism was a strong predictor, most commonly of both presence and severity (paranoid, schizotypal, borderline, histrionic, dependent, obsessive-compulsive). Interesting deviations from this include the fact that neuroticism only predicted severity in antisocial, but not presence, and presence but not severity in narcissistic features. Openness only predicted presence of Narcissistic PD, a result that is not easily interpreted. Finally, the Total PD symptom model is considered separately as the hurdle step predicting presence reflects a more stringent step between those with any PD and those with none at all. Here only neuroticism is predictive of both PD presence and severity, although conscientiousness and affiliation also predict severity.

The current study addresses an implicit assumption and likely limitation in much of the prior work linking personality traits and PD—specifically, although PD is not a normally distributed phenomenon in the population, it has consistently been modeled as such. First, we demonstrated that the basic assumptions of linear regression are violated and frequently result in the prediction of impossible values (i.e., negative counts). Second, we found that NB and NBH models do a comparable job of fitting the count distributions of PD symptoms and they cannot be distinguished quantitatively. Nevertheless, despite the flexibility of the NB distribution to account for a large proportion of individuals with zero values, this feature of the data is suggestive of distinct processes that are worth examining via two-step count models (Atkins & Gallop, 2007; Long, 1997). With these models interesting differences emerge across the two steps. When predicting individual diagnostic constructs, results suggest that neuroticism and affiliation are predictive of both PD presence and severity, whereas dominance is more often, and conscientiousness is exclusively, predictive of the presence of PD symptoms.

These results have implications for understanding the relationship between normal-range personality traits and PD. Issues related to both dimensionality and continuity in personality and PD have emerged from the proposed revisions for DSM-5. The proposed two-step diagnostic process for DSM-5 defines PD using a continuum of self/interpersonal impairment, with separate maladaptive personality traits provided to characterize phenotypic variation in the expression of an individual's core personality pathology. The models employed here are consistent with this approach, distinguishing between aspects of personality related to the presence of PD, and those related to its expression once present. Contrasting the results of the models predicting the count of all PD symptoms with the other model is also informative. The Total PD hurdle models are special in that they serve to represent general personality pathology, and the first step differentiates between those who have absolutely no pathology from those with any degree of pathology. The Total PD models suggest that only neuroticism differentiates those with any pathology from those without, which is consistent with research that shows it is an important predictor of myriad public health outcomes, psychological and otherwise (Lahey, 2009). Neuroticism also predicts severity along with low affiliation and low conscientiousness. This pattern of associations with severity was the same as that found by Hopwood and colleagues (2011) in a clinical sample. Findings here point to the fact that other variables besides normative traits (with the exception for a propensity to experience negative emotions) are responsible for the presence any PD, although other traits can characterize the variability in phenotypic expression of distinct PDs and overall PD severity. The implication is that normative traits alone are not ideal to differentiate normal from abnormal. The DSM-5 proposal to draw on self and interpersonal processes to define general personality pathology with traits used to clarify specific forms of PD is consistent with these results, although future research should incorporate the distinct predictors in these models to formally test the proposal.

A complementary way of understanding these results is that basic traits exert themselves at different levels of pathology. Limited prior research suggests that the level of traits across the spectrum of PD is nonlinear (O'Connor, 2005), indicating that normative traits may be more or less informative for distinguishing individuals at different levels of PD dimensions. Figure 2 illustrates this, continuing with NPD as the exemplar. Trait scores for those individuals without symptoms occur across the range of values, indicating that knowing someone's trait level without knowing their pathology is often diagnostically uninformative. For example, there are individuals at all levels of Dominance, including high levels, without any NPD symptoms. Yet, once there is any narcissistic pathology, rising severity is associated with increases in dominance. The opposite is true with neuroticism. Those without narcissistic pathology are lower on average, but once pathology is present, neuroticism is unrelated to severity. These fine-grained relationships suggest that PDs are not reducible to sums of basic traits, but are more complex in their structure of associations.

Figure 2. Scatter plots of personality trait scores and narcissistic personality disorder (NPD) features.

Several caveats must be mentioned. First, our present sample was more homogenous in age, education, and social class than the U.S. population. Second, given that the participants were selected from a population of first-year university students, the sample may have been somewhat censored for individuals affected by very severe PDs. However, the results from the linear regression models are highly consistent with prior work (e.g., Samuel & Widiger, 2008), suggesting generalizability. Additionally, the distributions of all psychiatric disorders in the LSPD sample are consistent with the U.S. population distribution (Kessler, Chiu, Demler, & Walters, 2005; Lenzenweger, 2008). Third, although some may feel that running individual models for each trait does not provide the most accurate picture of these relationships, the overwhelming majority of prior research adopts this approach, and therefore we employed it to provide a clear comparison for readers. Fourth, we should highlight that the linear regression models are still quite robust, and exhibit patterns of associations consistent with the NB models. Thus, we are confident that prior studies have accurately identified general relationships between traits and PD. However, a necessary next step in the empirical study of PD is to move beyond this level of analysis to elucidate the exact structure of these relationships in addition to identifying etiologic and mechanistic processes (e.g., underlying neurobehavioral systems, see Depue & Lenzenweger, 2005).

In summary, we examined the effect of varying the distribution of PD symptoms in regression models with personality traits. To appropriately model them requires the use of count distributions, and two-step count models provide opportunities to examine discontinuities in these relationships. In the past, modeling the distributions implemented here might have been more challenging, but a number of user-friendly statistical packages now include these as standard features. We used R, but Mplus, Stata, SAS, and SPSS can employ some or all of these distributions. When these approaches are adopted, a more refined picture emerges suggesting that PD is not merely the tail end of a continuous distribution of normal traits, and the traits associated with the presence of PD are not always those associated with increasing severity. Although we do not argue that these results are definitive and recognize that they should be replicated, we suggest that these analytic approaches are more appropriate, have the potential to elucidate some of the issues associated with continuity and discontinuity in personality and its pathology, and inform ongoing efforts to refine the diagnosis of PD.

¹ The distribution in Figure 1 is highly representative of each of the PDs in the LSPD.

² Full results available upon request.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed.), text revision. Washington, DC: Author.

Atkins, D. C., & Gallop, R. J. (2007). Rethinking how family researchers model infrequent outcomes: A tutorial on count regression and zero-inflated models. Journal of Family Psychology, 21, 726–735. doi:10.1037/0893-3200.21.4.726

Bagby, R. M., Costa, P. T., Jr., Widiger, T. A., Ryder, A. G., & Marshall, M. (2005). DSM-IV personality disorders and the five-factor model of personality: A multi-method examination of domain–and facet-level predictions. European Journal of Personality, 19, 307–324. doi:10.1002/per.563

Cameron, A. C., & Trivedi, P. K. (1998). Regression analysis of count data. Cambridge, United Kingdom: Cambridge University Press.

Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Erlbaum.

Coxe, S., West, S. G., & Aiken, L. S. (2009). The analysis of count data: A gentle introduction to Poisson regression and its alternatives. Journal of Personality Assessment, 91, 121. doi:10.1080/00223890802634175

Depue, R. A., & Lenzenweger, M. F. (2005). A neurobehavioral dimensional model of personality disturbance. In M. F.Lenzenweger & J. F.Clarkin (Eds.), Major theories of personality disorder (2nd ed., pp. 391–453). New York, NY: Guilford Press.

Hopwood, C. J., Malone, J. C., Ansell, E. B., Sanislow, C. A., Grilo, C. M., McGlashan, T. H., . . .Morey, L. C. (2011). Personality assessment in DSM-V: Empirical support for rating severity, style, and traits. Journal of Personality Disorders, 25, 305–320.

Kernberg, O. F. (1984). Severe personality disorders: Psychotherapeutic strategies. New Haven, CT: Yale University Press.

Kessler, R. C., Chiu, W. T., Demler, O., & Walters, E. E. (2005). Prevalence, severity, and comorbidity of the 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 617–627. doi:10.1001/archpsyc.62.6.617

Krueger, R. F., Eaton, N. R., Clark, L. A., Watson, D., Markon, K. E., . . .Livesley, W. J. (2011). Deriving an empirical structure of personality pathology for DSM-5. Journal of Personality Disorders, 25, 170–191. doi:10.1521/pedi.2011.25.2.170

Lahey, B. B. (2009). Public health significance of neuroticism. American Psychologist, 64, 241–256. doi:10.1037/a0015309

Lenzenweger, M. F. (1999). Stability and change in personality disorder features: The longitudinal study of personality disorders. Archives of General Psychiatry, 56, 1009–1015. doi:10.1001/archpsyc.56.11.1009

Lenzenweger, M. F. (2008). Epidemiology of personality disorders. Psychiatric Clinics of North America, 31, 395–403. doi:10.1016/j.psc.2008.03.003

Lenzenweger, M. F., & Clarkin, J. F. (2005). The personality disorders: History, development, and research issues. In M. F.Lenzenweger & J. F.Clarkin (Eds.), Major theories of personality disorder (2nd ed., pp. 1–42). New York, NY: Guilford.

Livesley, W. J., & Jang, K. L. (2005). Differentiating normal, abnormal, and disordered personality. European Journal of Personality, 19, 257–268. doi:10.1002/per.559

Long, J. S. (1997). Regression models for categorical and limited dependent variables. Thousand Oaks, CA: Sage Publications.

Loranger, A. W. (1999). International Personality Disorder Examination: DSM-IV and ICD-10 Interviews. Odessa, FL: Psychological Assessment Resources Inc.

Morey, L. C., Gunderson, J. G., Quigley, B. D., Shea, M. T., Skodol, A. E., . . .Zanarini, M. C. (2002). The representation of borderline, avoidant, obsessive-compulsive, and schizotypal personality disorders by the five-factor model. Journal of Personality Disorders, 16, 215–234. doi:10.1521/pedi.16.3.215.22541

Morey, L. C., Hopwood, C. J., Gunderson, J. G., Skodol, A. E., Shea, M. T., . . .McGlashan, T. H. (2007). Comparison of alternative models for personality disorders. Psychological Medicine, 37, 983–994. doi:10.1017/S0033291706009482

O'Connor, B. P. (2005). Graphical analysis of personality disorders in five-factor model space. European Journal of Personality, 19, 287–305. doi:10.1002/per.558

Pincus, A. L. (2005). A contemporary integrative interpersonal theory of personality disorders. In J.Clarkin & M.Lenzenweger (Eds.), Major theories of personality disorder (2nd ed., pp. 282–331). New York, NY: Guilford.

Samuel, D. B., & Widiger, T. A. (2008). A meta-analytic review of the relationships between the five-factor model and DSM-IV-TR personality disorders: A facet level analysis. Clinical Psychology Review, 28, 1326–1342. doi:10.1016/j.cpr.2008.07.002

Saulsman, L. M., & Page, A. C. (2004). The five-factor model and personality disorder empirical literature: A meta-analytic review. Clinical Psychology Review, 23, 1055–1085. doi:10.1016/j.cpr.2002.09.001

Skodol, A. E., Clark, L. A., Bender, D. S., Krueger, R. F., Morey, L. C., . . .Oldham, J. M. (2011). Proposed changes in personality and personality disorder assessment and diagnosis for DSM-5 Part I: Description and rationale. Personality Disorders: Theory, Research, and Treatment, 2, 4–22. doi:10.1037/a0021891

Trapnell, P. D., & Wiggins, J. S. (1990). Extension of the interpersonal adjective scales to include the big five dimensions of personality. Journal of Personality and Social Psychology, 59, 781–790. doi:10.1037/0022-3514.59.4.781

Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57, 307–333.

Widiger, T. A., Livesley, W. J., & Clark, L. A. (2009). An integrative dimensional classification of personality disorder. Psychological Assessment, 21, 243–255. doi:10.1037/a0016606

Widiger, T. A., & Trull, T. J. (2007). Plate tectonics in the classification of personality disorder: Shifting to a dimensional model. American Psychologist, 62, 71–83. doi:10.1037/0003-066X.62.2.71

Zeileis, A., Kleiber, C., & Jackman, S. (2008). Regression models for count data in R. Journal of Statistical Software, 27, 1–25.

Submitted: October 2, 2011 Revised: April 27, 2012 Accepted: May 22, 2012

Are Body Dissatisfaction, Eating Disturbance, and Body Mass Index Predictors of Suicidal Behavior in Adolescents? A Longitudinal Study

Acknowledgement: This study was supported by Maternal and Child Health Bureau Grant R40 MC 00319 from the Health Resources and Services Administration, Department of Health and Human Services; by Minnesota Obesity Center Grant P30DK 60456; and by National Institute of Mental Health Grant K02 MH 65919.

Suicide is a leading cause of death in adolescents (Goldsmith, 1986), and suicidal ideation and suicide attempts are common. In reviewing the published literature on suicidal behaviors during adolescence, Evans, Hawton, Rodham, and Deeks (2005) found mean lifetime rates of suicidal ideation of 29.9%; of respondents, 9.7% had reported suicide attempts. It appears that the frequency of suicidal behavior is typically underestimated by families (Breton, Tousignant, Bergeron, & Berthiaume, 2002; Klimes-Dougan, 1998) and might be underestimated by clinicians. Suicidal behavior results in substantial utilization of emergency care and hospitalization (Olfson, Gameroff, Marcus, Greenberg, & Shaffer, 2005).

Numerous correlates of suicidal behavior during adolescence have been identified. These correlates include family history of suicidal behavior (Brent & Mann, 2005), substance use disorders, early childhood maltreatment (Bridge, Goldstein, & Brent, 2006), and psychiatric disorders (Andrews & Lewinsohn, 1992; Brent et al., 1988; Fergusson, Beautrais, & Horwood, 2003; Kessler, Borges, & Walters, 1999). The psychiatric disorders implicated include, most prominently, mood disorders and eating disorders but also anxiety disorders and psychotic disorders. It appears that eating disorders may carry the highest suicide risk of any psychiatric disorder (Franko & Keel, 2006; Harris & Barraclough, 1994).

Although the link between diagnosable eating disorders and suicidal behavior has been recognized for some time, recent evidence suggests a correlation between suicidal behavior and body dissatisfaction, as well as disordered eating behaviors that fall short of the threshold for eating disorder, not otherwise specified, diagnosis (Ackard, Neumark-Sztainer, Story, & Perry, 2003; Crow, Eisenberg, Story, & Neumark-Sztainer, 2008; Rafiroiu, Sargent, Parra-Medina, Drane, & Valois, 2003). For example, in a school-based study of 13-year-olds, higher levels of body dissatisfaction predicted suicidal attempts over 2-year follow-up (Rodriguez-Cano, Beato-Fernandez, & Llario, 2006). Similarly, Crow et al. (2008) reported that unhealthy weight control behaviors (UWCB) and body dissatisfaction both correlated cross-sectionally with suicidal behavior in adolescents. In a sample of high-school-age adolescents, body image and, in particular, body attitudes and feelings predicted suicidal ideation (Brausch & Muehlenkamp, 2007).

Whether overweight and obesity are correlated with suicidal behavior has been unclear; some studies have found such a correlation (Carpenter, Hasin, Allison, & Faith, 2000; Falkner et al., 2001; Moore, Stunkard, & Srole, 1962), but others have not (Crow et al., 2008; Hallstrom & Noppa, 1981; Kittel, Rustin, Dramaix, deBacker, & Kornitzer, 1978; Neumark-Sztainer, Story, French, et al., 1997). Association of obesity and suicidal behavior in adolescents would represent a significant public health concern, given the marked increases in prevalence of obesity among adolescents.

Cross-sectional relationships have been described repeatedly, but the extent to which body dissatisfaction, disordered eating, or obesity is predictive of the occurrence of suicidal behaviors over time is not known. Identifying factors that are predictive of suicide risk is essential for treatment and prevention. In this study, we examined the longitudinal relationships between body dissatisfaction, weight control behaviors, weight status, and self-reported suicidal behavior using a nonclinical sample of adolescents and young adults. To expand on previous work, we examined a larger group of both male and female adolescents over a longer period of time at multiple ages. We hypothesized that body dissatisfaction, weight control behaviors, and obesity would be positively associated with self-reported suicidal ideation and suicide attempts at 5-year follow-up.

Participants

Data for this analysis came from Project EAT-II (Neumark-Sztainer, Wall, Eisenberg, Story, & Hannan, 2006; Neumark-Sztainer, Wall, Guo, et al., 2006). Project EAT-II was a follow-up study of adolescents who had participated in Project EAT (Neumark-Sztainer, Story, Hannan, & Croll, 2002; Neumark-Sztainer, Story, Hannan, Perry, & Irving, 2002), which had examined dietary intake, weight status, eating behaviors, and socioenvironmental and demographic correlates in a large, ethnically diverse study population. Project EAT involved 4,746 Minnesota middle and high school students who were initially surveyed during 1998–1999. In Project EAT-II, we aimed to recontact all of the original participants 5 years after the initial study. A total of 1,074 (22.6%) of the original cohort was lost to follow-up, mainly because no address was found at follow-up (n = 591) or contact information was missing at Time 1 (n = 411). Of the original participants, 3,672 were contacted by mail in 2003–2004 and 2,516 (68.4% of those contacted; 1,386 female and 1,130 male) completed the Project EAT-II survey. The follow-up sample was more likely to be White and at higher socioeconomic status (SES); likely, this distribution was due to higher mobility in the lower SES and non-White groups, which were heavily represented in the original EAT sample. We used population weights that reflected the original sample in all analyses to address this shortcoming. About one third of the sample was in the high-school-age cohort (mean age = 17.2 years), and two thirds of the sample was in the young adult group (mean age = 20.4 years). The sample was ethnically diverse and comprised Asian Americans (19.2%), African Americans (18.7%), Hispanics/Latinos (5.8%), Native Americans (3.6%), and individuals of mixed race (4.3%); the remainder (48.4%) were Whites. A wide socioeconomic distribution was seen: Most participants fell in the lower middle, middle, or upper middle SES categories; 17.4% were of low SES, and 13.8% were of high SES. Further details of Project EAT-II are available elsewhere (Neumark-Sztainer, Wall, Eisenberg, et al., 2006; Neumark-Sztainer, Wall, Guo, et al., 2006). All study procedures for both Project EAT and Project EAT-II were approved by the University of Minnesota's Human Subject Committee.

Measures

Weight status

At Time 1, weight and height were measured by trained Project EAT staff in school settings. Weight was measured in light clothing, and height was measured without shoes. Body mass index (BMI; weight [kg]/height squared [m²]) was calculated, and participants were classified as being underweight (<15th BMI percentile), average weight (15th–85th BMI percentile), moderately overweight (85th–95th BMI percentile), or very overweight (≥95th BMI percentile), according to gender- and age-based cutpoints recommended by the Centers for Disease Control and Prevention (Kuczmarski et al., 2000).

Body satisfaction

Participants completed a 10-item scale (Pingitore, Spring, & Garfield, 1997) on which satisfaction with separate body parts and characteristics was rated from 1 (very dissatisfied) to 5 (very satisfied). For example, an item on the scale asks “How satisfied are you with your stomach?” A score ranging from 10 to 50 was created; greater scores indicated greater levels of body satisfaction (Pingitore et al., 1997). Cronbach's alpha at Time 2 was .92 for young women and .93 for young men for the composite score. In this study, test–retest reliability was acceptable, with Pearson correlations of .68–.77 in a racially diverse subgroup of 252 7th- and 10th-grade participants. The median score was 35; the top quartile cutoff was 41, and the lowest quartile cutoff was 28. In the current analyses, the lowest quartile was considered to have “body dissatisfaction.”

Suicidal thoughts and behaviors

Students reported on both suicide attempts and suicidal ideation. The survey questions used included “Have you ever thought about killing yourself?” and “Have you ever tried to kill yourself?” Response options included “no,” “yes, in the past year,” and “yes, more than a year ago.”

Those who reported past-year suicidal ideation or attempts at Time 2 were categorized as cases, regardless of their Time 1 suicidal status. Those who reported suicidal ideation or attempts “more than a year ago” at Time 2 but no suicidal reports at Time 1 were also categorized as cases. All others were considered to have no suicidal thoughts or behaviors at Time 2. In this way, we were able to identify suicidal ideation and attempts that had occurred after the assessment of Time 1 predictor variables. Two-week test–retest Spearman correlations at Time 1 were .78 (ideations) and .80 (attempts).

UWCB

The UWCB assessed included (1) taking diet pills, (2) “making myself vomit,” (3) using laxatives, (4) using diuretics, (5) fasting, (6) eating very little food, (7) using a food substitute (powder/special drink), (8) skipping meals, and (9) smoking more cigarettes. For each item, participants were asked to answer “yes” or “no” to “Have you done any of the following things in order to lose weight or keep from gaining weight during the past year?” Participants who endorsed any of Items 1–4 were classified as using extreme weight control behaviors (EWCB); participants who endorsed any of Items 5–9 were considered to have UWCB.

Depressive symptoms

We used a six-item scale to assess depressive symptoms (Kandel & Davies, 1982). At Time 2, the scale had a Cronbach's alpha of .80 for young men and .81 for young women. For the current analyses, those who scored in the highest quartile were considered to have high levels of depressive symptoms.

Demographic variables

SES was categorized into five levels with an algorithm based on highest educational level completed by either parent plus eligibility for free or reduced-price lunch, eligibility for public assistance, and parental employment status (Neumark-Sztainer, Story, Hannan, & Croll, 2002). Race/ethnicity was assessed by asking “Do you think of yourself as (a) White, (b) Black or African American, (c) Hispanic or Latino, (d) Asian American, (e) Hawaiian or Pacific Islander, (f) American Indian or Native American, or (g) other race?” Participants were asked to check all that applied. For purposes of this analysis, all non-Whites were grouped together. Analyses were adjusted for race/ethnicity on the basis of this grouping.

Data Analysis

Data were weighted to adjust for differential response rates with the response propensity method (Little, 1986), in which the inverse of the estimated probability that an individual would respond at Time 2 was used as the weight. Estimates were therefore generalizable to the population represented by the original Time 1 Project EAT sample.

We used separate multiple logistic regression analyses to estimate odds ratios for Time 2 suicidal ideation and attempts. In unadjusted analyses, four Time 1 weight-related variables (weight status, body dissatisfaction, EWCB, and UWCB) were entered simultaneously. A second model adjusted for race, SES, and age group. This model was further adjusted for high depressive symptoms at Time 2, given that depression has been associated with suicidal behavior.

Weight distribution, frequencies of EWCB and UWCB at Time 1, suicidal behavior at Time 2, and participant characteristics are shown in Table 1. UWCB were endorsed by the majority of young women (57.0%) and a large percentage of the young men (31.1%). EWCB were less common (12.9% of young women and 3.9% of young men). Suicidal ideation was reported by 21.6% of young women (12.6% in the past year; 8.9% more than 1 year earlier) and by 15.2% of young men (8.3% in the past year; 7.0% more than 1 year earlier) at Time 2, whereas suicide attempts were reported by 8.7% of young women and 3.5% of young men at Time 2.

Demographics and Key Variables by Gender

The unadjusted and adjusted relationships between Time 1 weight-related variables (weight status, body dissatisfaction, EWCB, and UWCB) and Time 2 suicidal ideation are shown in Table 2. For young women, EWCB were predictive of later suicidal ideation (odds ratio [OR] = 1.98, 95% confidence interval [CI] = 1.34–2.93). These ORs remained elevated even after we had adjusted for demographic variables and Time 2 depressive symptoms (OR = 1.79, 95% CI = 1.19–2.71). In contrast, among young men, the relationship between EWCB and suicidal ideation was not statistically significant. Furthermore, UWCB, body dissatisfaction, and weight status at baseline each failed to predict suicidal ideation at follow-up for male or female participants.

Weight Status, Body Dissatisfaction, and Weight Control Behaviors at Time 1 and Suicidal Ideation at Time 2

Table 3 shows the relationship between Time 1 weight status, body dissatisfaction, weight control behaviors, and Time 2 reported suicide attempts. Similarly, EWCB in young women was associated with a significantly elevated OR for suicide attempts (OR = 2.53, 95% CI = 1.53–4.18). These ORs remained elevated after we had adjusted for demographic variables and Time 2 depressive symptoms (OR = 2.41, 95% CI = 1.43–4.07). No association between suicide attempts and EWCB was found among young men. As with suicidal ideation, UWCB, body dissatisfaction, and weight status were not significantly associated with suicide attempts among either male or female participants over 5-year follow-up.

Weight Status, Body Dissatisfaction, and Weight Control Behaviors at Time 1 and Suicide Attempts at Time 2

The results of this study show that, in young women but not in young men, EWCB at baseline were predictive of suicidal ideation and suicide attempts at 5-year follow-up independent of depressive symptoms. Contrary to our hypotheses, body dissatisfaction, UWCB, and weight status were not predictive of suicidal behavior 5 years later in male or female participants.

These findings are consistent with the results of several previous studies that have shown an association between both syndromal eating disorders (Harris & Barraclough, 1994) and limited eating disorder symptoms and suicidal behaviors (Crow et al., 2008; Miotto, De Coppi, Frezze, & Preti, 2003). Previous studies were cross-sectional, however, and the current study indicates that EWCB are predictive of suicidal ideation and suicide attempts over time. Our results suggest that EWCB might be a risk factor or risk marker for later suicidality. Although the rates of suicidal ideation and attempts endorsed by participants were high, they were in the range of those reported in other community-based studies (Centers for Disease Control, 2000; Kessler et al., 1999).

Differential associations between EWCB and suicidal thoughts and behaviors were found between young men and young women. The smaller sample size of young men endorsing EWCB (as well as suicidal behaviors) and the resulting limited power to detect such efforts may explain this finding, given that the ORs observed were similar in male and female participants (but confidence intervals were larger in the former). An alternative explanation may be that differing societal attitudes regarding the importance of weight and shape make EWCB more emotionally salient for women. If this were the case, EWCB might have had more psychopathological consequences for young women than for young men.

Results from the current study differ from previous cross-sectional work, which has found an association between body dissatisfaction and suicidal behavior in adolescents (Crow et al., 2008; Miotto et al., 2003). The reasons for these divergent findings are unclear. Perhaps body dissatisfaction is associated with psychosocial distress that might have short-term but not long-term links to suicidal behavior. This study, with reassessment after 5 years, did not examine short-term relationships between these variables. Alternatively, body dissatisfaction and UWCB may be so common among young women at present as to be considered normative; perhaps only more severe psychopathology (e.g., weight control behaviors defined as “extreme”: purging, laxative use, and diuretic use) is predictive of suicidal behavior. This relationship might operate at the personality trait level. For example, underlying impulsivity has been associated with both disordered eating and suicidal behavior. Recent work has suggested that altered serotonin function may be correlated with the co-occurrence of self-harm, impulsivity, and bulimic symptoms (Steiger et al., 2001).

The results of this study further our understanding of the relationship between body weight and suicidal behavior. The finding that weight was unrelated to suicidal behavior is consistent with some prior work in this area (Crow et al., 2008; Hallstrom & Noppa, 1981; Kittel et al., 1978; Mukamal, Kawachi, Miller, & Rimm, 2007; Neumark-Sztainer, Story, Resnick, & Blum, 1997) but contradicts other studies (Carpenter et al., 2000; Falkner et al., 2001; Neumark-Sztainer, Story, Resnick, & Blum, 1997). Although the cross-sectional prior studies have been mixed, it is noteworthy that the two prospective studies examining this question (Mukamal et al., 2007, and the current study) have not found higher BMI to be predictive of suicide.

There are a number of strengths and some limitations to the current project. The sample size was large and diverse, in terms of gender, ethnicity, and SES. In addition, the use of a prospective design with 5-year follow-up broadened our understanding of the relationship between EWCB and later suicidal behavior. Limitations include the lack of extensive detail regarding suicidal ideation and suicide attempts. It would be helpful to have more information in this regard, particularly on the timing, frequency, and severity of suicidal behaviors over the 5-year period. There is evidence that varying the method of suicide assessment leads to variation in the reported rate of suicidal behavior (Prinstein, Nock, Spirito, & Grapentine, 2001). Similarly, more detailed measures of body dissatisfaction and disordered eating behaviors/attitudes would have been helpful and might have yielded different findings. Attrition at 5-year reassessment is another limitation. Finally, although our study was prospective, it nonetheless entailed some degree of retrospective recall at the 5-year reassessment point.

These findings have important implications for future research and practice. They help clarify the link between disordered eating and suicidal behavior. These findings further emphasize the importance of screening for EWCB. Levels of disordered eating that may fall short of diagnostic thresholds have been viewed as less serious, but such disordered eating is tied to psychosocial and physical morbidity and, this study suggests, is associated with an increased risk of suicidal behavior. It is unclear from the current study whether EWCB lead to suicidal thoughts/behaviors through some underlying mechanism or rather are a sign of more global distress and a risk marker for suicide risk. In either case, the use of EWCB by an adolescent suggests a need for careful monitoring of the adolescent's mental health. Future research should examine this issue over even longer periods of observation. In addition, it would be of interest to know whether the risk for adolescent suicidal behavior conferred by EWCB carries over into adulthood. If the link between EWCB and suicidal behavior is confirmed, efforts directed at preventing the onset of disordered eating might diminish the risk of subsequent suicidal behavior.

Ackard, D. M., Neumark-Sztainer, D., Story, M., & Perry, C. (2003). Overeating among adolescents: Prevalence and associations with weight-related characteristics and psychological health. Pediatrics, 111(1), 67–74.

Andrews, J. A., & Lewinsohn, P. M. (1992). Suicidal attempts among older adolescents: Prevalence and co-occurrence with psychiatric disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 31(4), 655–662.

Brausch, A. M., & Muehlenkamp, J. J. (2007). Body image and suicidal ideation in adolescents. Body Image, 4, 207–212.

Brent, D. A., & Mann, J. J. (2005). Family genetic studies, suicide, and suicidal behavior. American Journal of Medical Genetics Part C, Seminars in Medical Genetics, 133(1), 13–24.

Brent, D. A., Perper, J. A., Goldstein, C. E., Kolko, D. J., Allan, M. J., Allman, C. J., et al. (1988). Risk factors for adolescent suicide. A comparison of adolescent suicide victims with suicidal inpatients. Archives of General Psychiatry, 45(6), 581–588.

Breton, J. J., Tousignant, M., Bergeron, L., & Berthiaume, C. (2002). Informant-specific correlates of suicidal behavior in a community survey of 12- to 14-year-olds. Journal of the American Academy of Child and Adolescent Psychiatry, 41(6), 723–730.

Bridge, J. A., Goldstein, T. R., & Brent, D. A. (2006). Adolescent suicide and suicidal behavior. Journal of Child Psychology and Psychiatry, 47(3–4), 372–394.

Carpenter, K. M., Hasin, D. S., Allison, D. B., & Faith, M. S. (2000). Relationships between obesity and DSM–IV major depressive disorder, suicide ideation, and suicide attempts: Results from a general population study. American Journal of Public Health, 90(2), 251–257.

Centers for Disease Control. (2000). Youth risk behavior surveillance: United States, 1999. Morbidity and Mortality Weekly Report Surveillance Summaries, 49, 1–96.

Crow, S., Eisenberg, M. E., Story, M., & Neumark-Sztainer, D. (2008). Suicidal behavior in adolescents: Relationship to weight status, weight control behaviors, and body dissatisfaction. International Journal of Eating Disorders, 41(1), 82–87.

Evans, E., Hawton, K., Rodham, K., & Deeks, J. (2005). The prevalence of suicidal phenomena in adolescents: A systematic review of population-based studies. Suicide & Life Threatening Behavior, 35(3), 239–250.

Falkner, N. H., Neumark-Sztainer, D., Story, M., Jeffery, R. W., Beuhring, T., & Resnick, M. D. (2001). Social, educational, and psychological correlates of weight status in adolescents. Obesity Research, 9(1), 32–42.

Fergusson, D. M., Beautrais, A. L., & Horwood, L. J. (2003). Vulnerability and resiliency to suicidal behaviours in young people. Psychological Medicine, 33(1), 61–73.

Franko, D. L., & Keel, P. K. (2006). Suicidality in eating disorders: Occurrence, correlates, and clinical implications. Clinical Psychology Review, 26(6), 769–782.

Goldsmith, R. E. (1986). Dimensionality of the Rosenberg Self-Esteem Scale. Journal of Social Behavior and Personality, 1, 253–264.

Hallstrom, T., & Noppa, H. (1981). Obesity in women in relation to mental illness, social factors and personality traits. Journal of Psychosomatic Research, 25(2), 75–82.

Harris, E. C., & Barraclough, B. (1994). Suicide as an outcome for mental disorders. A meta-analysis. British Journal of Psychiatry, 170, 205–228.

Kandel, D. B., & Davies, M. (1982). Epidemiology of depressive mood in adolescents: An empirical study. Archives of General Psychiatry, 39(10), 1205–1212.

Kessler, R. C., Borges, G., & Walters, E. E. (1999). Prevalence of and risk factors for lifetime suicide attempts in the National Comorbidity Survey. Archives of General Psychiatry, 56(7), 617–626.

Kittel, F., Rustin, R. M., Dramaix, M., deBacker, G., & Kornitzer, M. (1978). Psycho-socio-biological correlates of moderate overweight in an industrial population. Journal of Psychosomatic Research, 22, 145–158.

Klimes-Dougan, B. (1998). Screening for suicidal ideation in children and adolescents: Methodological considerations. Journal of Adolescence, 21(4), 435–444.

Kuczmarski, R. J., Ogden, C. D., Grummer-Strawn, L. M., Flegal, K. M., Guo, S. S., Wei, R., et al. (2000). CDC growth charts: United States. Advance Data, 314, 1–27.

Little, R. (1986). Survey nonresponse adjustments for estimates of means. International Statistical Review, 54(2), 137–139.

Miotto, P., De Coppi, M., Frezze, M., & Preti, A. (2003). Eating disorders and suicide risk factors in adolescents: An Italian community-based study. Journal of Nervous and Mental Disorders, 191(7), 962–966.

Moore, M. E., Stunkard, A., & Srole, L. (1962). Obesity, social class, and mental illness. Journal of the American Medical Association, 181, 962–966.

Mukamal, K. J., Kawachi, I., Miller, M., & Rimm, E. B. (2007). Body mass index and risk of suicide among men. Archives of Internal Medicine, 167, 468–475.

Neumark-Sztainer, D., Story, M., French, S. A., Hannan, P. J., Resnick, M. D., & Blum, R. W. (1997). Psychosocial concerns and health-compromising behaviors among overweight and nonoverweight adolescents. Obesity Research, 5(3), 237–249.

Neumark-Sztainer, D., Story, M., Hannan, P. J., & Croll, J. (2002). Overweight status and eating patterns among adolescents: Where do youths stand in comparison with the healthy people 2010 objectives?American Journal of Public Health, 92(5), 844–851.

Neumark-Sztainer, D., Story, M., Hannan, P. J., Perry, C. L., & Irving, L. M. (2002). Weight-related concerns and behaviors among overweight and nonoverweight adolescents: Implications for preventing weight-related disorders. Archives of Pediatric and Adolescent Medicine, 156(2), 171–178.

Neumark-Sztainer, D., Story, M., Resnick, M. D., & Blum, R. W. (1997). Psychosocial concerns and weight control behaviors among overweight and nonoverweight Native American adolescents. Journal of the American Diatetic Association, 97(6), 598–604.

Neumark-Sztainer, D., Wall, M. M., Eisenberg, M. E., Story, M., & Hannan, P. J. (2006). Overweight status and weight control behaviors in adolescents: Longitudinal and secular trends from 1999–2004. Preventative Medicine, 43(1), 52–59.

Neumark-Sztainer, D., Wall, M. M., Guo, J., Story, M., Haines, J., & Eisenberg, M. E. (2006). Obesity, disordered eating, and eating disorders in a longitudinal study of adolescents: How do dieters fare five years later?Journal of the American Dietetic Association, 106(4), 559–568.

Olfson, M., Gameroff, M. J., Marcus, S. C., Greenberg, T., & Shaffer, D. (2005). Emergency treatment of young people following deliberate self-harm. Archives of General Psychiatry, 62(10), 1122–1128.

Pingitore, R., Spring, B., & Garfield, D. (1997). Gender differences in body satisfaction. Obesity Research, 5(5), 402–409.

Prinstein, M. J., Nock, M. K., Spirito, A., & Grapentine, W. L. (2001). Multimethod assessment of suicidality in adolescent psychiatric inpatients: Preliminary results. Journal of the American Academy of Child and Adolescent Psychiatry, 40(9), 1053–1061.

Rafiroiu, A. C., Sargent, R. G., Parra-Medina, D., Drane, W. J., & Valois, R. F. (2003). Covariations of adolescent weight-control, health-risk and health-promoting behaviors. American Journal of Health Behavior, 27(3), 3–14.

Rodriguez-Cano, T., Beato-Fernandez, L., & Llario, A. B. (2006). Body dissatisfaction as a predictor of self-reported suicide attempts in adolescents: A Spanish community prospective study. Journal of Adolescent Health, 38(6), 684–688.

Steiger, H., Koerner, N., Engelberg, M. J., Israel, M., Ng Ying Kin, N. M., & Young, S. N. (2001). Self-destructiveness and serotonin function in bulimia nervosa. Psychiatry Research, 103(1), 15–26.

Submitted: August 8, 2007 Revised: April 14, 2008 Accepted: April 25, 2008

Behavioral Activation for Depressed Breast Cancer Patients: The Impact of Therapeutic Compliance and Quantity of Activities Completed on Symptom Reduction

Acknowledgement: This research was supported by Susan G. Komen for the Cure Research Grant BCTR0706709 awarded to Derek R. Hopko.

Behavioral activation (BA) is the therapeutic process of increasing overt behaviors to facilitate exposure to reinforcing environmental contingencies and subsequent reductions in depression (Hopko, Lejuez, Ruggiero, & Eifert, 2003). BA has evolved significantly in recent decades (Dimidjian, Barrera, Martell, Munoz, & Lewinsohn, 2011; Hopko, Ryba, McIndoo, & File, in press), and with growing empirical support, BA is now considered an empirically validated intervention and is an appealing treatment option for depression across a range of settings (Cuijpers, van Straten, & Warmerdam, 2007; Ekers, Richards, & Gilbody, 2008; Sturmey, 2009). There is still much to be learned about the process of change in BA, however, and no systematic longitudinal research has explored whether increased activation and environmental reinforcement are in fact central mediators of change. Accordingly, in the context of a recently completed randomized trial of BA and problem-solving therapy for depressed breast cancer patients (Hopko, Armento, et al., 2011), the current study was designed to more clearly explicate the relations among structured BA, environmental reward, and depression.

BA is rooted in behavioral models of depression that implicate decreased response-contingent positive reinforcement (RCPR) for nondepressive behavior as the causal factor in eliciting depression (Ferster, 1973; Lewinsohn, 1974). This reduction in RCPR is attributable to a decrease in the number and range of reinforcing stimuli available to an individual for such behavior, a lack of skill in obtaining reinforcement, and/or an increased frequency of punishment (Lewinsohn, 1974). This view suggests that depressed behavior results from a combination of reinforcement for depressed behavior and a lack of reinforcement or even punishment for more healthy alternative behaviors (Ferster, 1973; Hopko, Lejuez, et al., 2003; Lewinsohn, 1974). As a result, depressed individuals often experience significant behavioral inhibition and avoidance behaviors, the central target of contemporary behavioral treatments for depression: behavioral activation (BA; Martell, Addis, & Jacobson, 2001) and the brief behavioral activation treatment for depression (BATD; Lejuez, Hopko, & Hopko, 2001; revised version, BATD–R; Lejuez, Hopko, Acierno, Daughters, & Pagoto, 2011). Although BA approaches commonly are based on behavioral principles of reinforcement and functional assessment, specific treatment strategies differ across interventions (Kanter et al., 2010), with activity monitoring and activity scheduling being the two constant features of both BA protocols (Addis & Martell, 2004; Lejuez et al., 2001). The BA method (Martell et al., 2001) incorporates strategies of change including identification of avoidance patterns, teaching functional assessment of behavior, guided activity, mental rehearsal, periodic distraction, mindfulness training, rumination-cued activation, and skills training. Alternatively, BATD focuses on functional assessment of depressed behavior, identification of activities based on individualized goals and a life values assessment, and structured systematic activation using a hierarchy of activities. These contemporary versions of BA are considered more idiographic than conventional behavior therapy (Hopko, Lejuez, et al., 2003). Perhaps most important, BA and BATD moved away from the assumption that pleasant activities have reinforcing properties and instead focus on environmental contingencies maintaining depressed behavior, unique value systems, and the targeting of avoidance through an emotional acceptance and behavior change paradigm.

Compelling treatment outcome research suggests BA strategies have broad applicability across a wide range of settings and clinical populations. In one of the more compelling studies, BA was comparable to antidepressant medication and superior to cognitive therapy in treating severe depression (Dimidjian et al., 2006), results that were maintained at 2-year follow-up (Dobson et al., 2008). BA also has been effectively used with depressed patients in various settings and among individuals with coexistent medical problems such as HIV, cancer, brain trauma, and obesity, as well as coexistent psychiatric problems such as anxiety disorders, schizophrenia, and borderline personality disorder (see Hopko et al., in press, for a comprehensive review). Three independent meta-analyses support the efficacy of BA in treating depression (Cuijpers et al., 2007; Ekers et al., 2008; Mazzucchelli, Kane, & Rees, 2009).

BA models of depression attribute affective change to increases in RCPR for healthy behaviors. Several studies support this model, demonstrating a relationship between depressed mood and the frequency of pleasant activities and increased reinforcement (Grosscup & Lewinsohn, 1980; Hopko, Armento, et al., 2003; Lewinsohn & Libet, 1972; Lewinsohn & Shaffer, 1971; Lewinsohn & Shaw, 1969; MacPhillamy & Lewinsohn, 1974). In a study examining the relation of activation and depression using daily diary methods, self-reported depression was inversely related to general activity level as well as the level of self-reported reward or pleasure obtained through engaging in overt behaviors (Hopko, Armento, et al., 2003). Another recent study showed depressed college students engaged less frequently in social, physical, and educational behaviors (Hopko & Mullane, 2008). Although these cross-sectional data are significant, there continues to be a paucity of systematic longitudinal research that adequately demonstrates the process by which therapeutic effects are obtained in BA. The lack of research documenting a causal relationship between exposure to reinforcers and depression reduction was partially due to the lack of available statistical meditational analyses during early stages of BA research. Although three recent studies render support for a relationship between activation and reduced depression via meditational effects of environmental reinforcement (Carvalho & Hopko, 2011; Carvalho, Trent, & Hopko, 2011; Ryba & Hopko, 2012), none of these studies incorporated a sophisticated longitudinal design that allowed examination of a definitive temporal relationship between BA and reduced depressive affect.

A number of theoretical and empirical questions pertaining to BA remain unanswered and warrant continued scientific investigation to allow researchers to better conceptualize and refine behavioral treatments for depression. For example, although BA researchers and theorists implicate activity scheduling (and subsequent exposure to environmental reward) as the primary active component in BA, no systematic research has supported this hypothesis. Indeed, some versions of BA include many treatment components (e.g., skills training, rumination-cued activation, cognitive rehearsal) that raise speculation of whether alternate change mechanisms account for favorable treatment outcomes (Addis & Martell, 2004; Martell et al., 2001). Second, at this stage of BA research, it is largely unclear whether the specific quantity of behaviors or the proportion completed (i.e., treatment compliance) is more essential toward conceptualizing positive treatment outcome. Clarification of the process of change in BA would assist mental health providers and facilitate further BA treatment refinement by identifying components of BA that account for significant outcome variance. To address these issues, this study examined relationships among the quantity of activities completed, proportion of activities completed (i.e., therapeutic compliance), environmental reward, and depression reduction in breast cancer patient treated with eight sessions of BATD. Study hypotheses were as follows:
Hypothesis 1: As BATD progressed, the overall quantity of activities assigned was predicted to increase.
Hypothesis 2: As BATD progressed, treatment compliance (i.e., proportion of activities completed) was predicted to increase.
Hypothesis 3: The proportion of activities completed and reduced depression would be mediated by increased environmental reward.
Hypothesis 4: The quantity of activities completed and reduced depression would be mediated by increased environmental reward.
Hypothesis 5: Individuals completing a greater quantity of activities and greater proportion of assigned activities were predicted to achieve treatment response and remission at posttreatment.

Participants

Participants were 23 breast cancer patients with a diagnosis of major depression who were treated at the University of Tennessee Medical Center’s Cancer Institute as part of a randomized clinical trial (Hopko, Armento, et al., 2011). All participants provided informed consent prior to study enrollment. Patients were recruited through clinic screening, clinic brochures, and oncologist referral. Patients interested in study participation completed a pretreatment diagnostic assessment that included the Anxiety Disorders Interview Schedule–IV (ADIS–IV; Brown, Di Nardo, & Barlow, 1994) and self-report instruments outlined in the following section. Advanced doctoral students conducted psychological assessments and were supervised by the principal investigator (DH) in the context of audiotape review and discussion, resulting in a consensus diagnosis. Individuals were eligible to participate if they were older than age 18, had been diagnosed with breast cancer, had a principle (and primary) diagnosis of major depression, and were not psychotic or cognitively impaired. The clinical trial included 80 depressed breast cancer patients, of which 42 were assigned to BATD. For the purposes of this study, only BATD patients who completed and returned all behavioral activation monitoring logs were included (n = 23).

The majority of these patients were White (91.3%); 8.7% were African American. The mean age was 57 years (SD = 11.3), and the average length of education was 15.2 years (SD = 2.2). Marital status was as follows: married (56.5%), single (21.7%), divorced (17.4%), and separated (4.3%). As reported in the randomized trial (Hopko, Armento, et al., 2011), for purposes of generalizability, antidepressant and antianxiety medication usage was not exclusionary. In this sample, 11 patients (48%) were prescribed antidepressant or antianxiety medication, and all had been stabilized at a consistent dosage for 8 weeks prior to study assessment, with no variations in medications or dosages throughout the trial (as reported by patients prior to each BATD session). The average time since breast cancer diagnosis was 2.7 years (SD = 1.9), and average tumor size was 2.57 cm (SD = 0.5). Patients of all cancer stages were included: Stage 0 (lobular carcinoma in situ or ductal carcinoma in situ: n = 7, 30%); Stage 1 (n = 7, 30%); Stage 2 (n = 5, 22%); Stage 3 (n = 3, 14%); and Stage 4 (n = 1, 4%). In terms of cancer treatment, 87% of patients had surgery (i.e., lumpectomy, mastectomy), 64% had chemotherapy, and 70% had radiation treatment. All patients in this sample had successfully completed their respective cancer treatment regimen prior to commencing BATD. Important insofar as assessing representation of the entire BATD sample (n = 42), other than failing to maintain monitoring logs, a series of analyses of variance for continuous variables and chi-square analyses for categorical variables indicated that the study sample (n = 23) and holdout sample (n = 19) did not statistically differ on any demographic, cancer-related, or psychological variables, including treatment response, χ²(1) = 2.29, p = .20, and remission, χ²(1) = 0.02, p = .99, rates following BATD (see Hopko, Armento, et al., 2011), as well as pretreatment depression severity, anxiety severity, social support, environmental reward, or self-reported bodily pain. For pretreatment depression severity, the study sample and holdout group did not differ on either the Beck Depression Inventory–II (BDI–II; Beck, Steer, & Brown, 1996), t(78) = 0.38, p = .71, or the Hamilton Rating Scale for Depression (Hamilton, 1960), t(78) = 0.87, p = .39. The two samples also did not differ as a function of the number of coexistent anxiety disorders, t(78) = 0.75, p = .46.

Assessment Measures

Behavioral monitoring

In BATD, the master activity log is used by the clinician to track weekly patient progress. All activities are listed on the master activity log, including (a) the number of times the patient eventually would like to complete the activity in a 1-week period (i.e., ideal frequency) and (b) the duration of the activity. In the initial session of BATD, fewer activities are monitored, with the number of activities progressively increasing in subsequent weeks as a function of patient success. On the behavioral checkout that is maintained by the patient, the frequency and duration of goals for each week also are listed, and the patient records which behaviors are completed on a daily basis. The patient returns the behavioral checkout to the clinician each week, and the information is transferred to the master activity log. If the patient has achieved (or exceeded) goals, the clinician likely will increase the frequency and/or duration for the following week (assuming the patient has not met the ideal goal). If a behavioral assignment was not completed, the clinician and patient decide if the assignment was reasonable or whether it was excessive or improbable given the patient’s abilities. In the former case, the goal might be the same for the next week or its importance (i.e., consistency with life values) re-evaluated. In the latter case, the clinician and patient might strongly consider reducing the weekly goal, and potentially the ideal goal. The rate at which new activities are added can occur slowly or rapidly across patients and generally is determined based on individual circumstances.

Beck Depression Inventory–II

The BDI–II consists of 21 items rated on a 4-point Likert scale. The BDI–II has excellent reliability and validity in depressed adults (Beck et al., 1996). The psychometric properties of the BDI–II also have been studied in cancer patients and a medical care sample, with strong predictive validity as it pertains to a diagnosis of clinical depression, strong internal consistency (α = .94), and adequate item-total correlations (range = .54–.74; Arnau, Meagher, Norris, & Bramson, 2001; Katz, Kopek, Waldron, Devins, & Thomlinson, 2004; α = .84; range = 14–60; M = 27.0, SD = 8.5, for the present study).

Environmental Reward Observation Scale

The EROS (Armento & Hopko, 2007) is a 10-item measure assessing exposure to environmental rewards deemed essential for increasing response-contingent positive reinforcement (RCPR; Lewinsohn, 1974). RCPR is defined as positive or pleasurable outcomes that follow behaviors; the outcomes may be extrinsic (e.g., social, monetary) or intrinsic (e.g., physiological, feeling of achievement) and increase the likelihood the behaviors will occur in the future. Decreased RCPR is a central predictor of increased depression (Lewinsohn, 1974). Higher scores on the EROS suggest increased environmental reward. Sample items include “The activities I engage in have positive consequences,” and “Lots of activities in my life are pleasurable.” Based on psychometric research with three independent college samples, the EROS has strong internal consistency (α = .85–.86) and excellent test–retest reliability (r = .85) and correlates strongly with other psychometrically sound measures of depression (r = from –.63 to –.69) and anxiety (Armento & Hopko, 2007). In this study, internal consistency was adequate (α = .78; M = 22.7, SD = 4.6).

Behavioral Activation Therapy for Depression (BATD)

BATD focuses on increasing overt behaviors to bring patients into contact with reinforcing environmental contingencies and corresponding improvements in thoughts, mood, and quality of life (Hopko, Lejuez, et al., 2003). Within BATD (Hopko & Lejuez, 2007; Lejuez et al., 2001, 2011), the process of increasing RCPR follows the basic principles of extinction, shaping, fading, and in vivo exposure (Hopko, Lejuez, et al., 2003). Initial sessions involved assessing the function of depressed behavior, establishing patient rapport, motivational exercises focused on the pros and cons of behavioral change, depression and breast cancer psychoeducation, and introduction of the treatment rationale. Within BATD, systematically increased activity is a necessary precursor toward the reduction of overt and covert depressed behavior. Patients began with a self-monitoring (daily diary) exercise to examine already occurring daily activities to provide a baseline measurement and ideas of activities to target during treatment. Patients were asked to keep a daily diary during 4 days of the week and to monitor their primary overt behaviors at half-hour intervals (from 8:00 a.m. to 2:00 a.m.). For each behavior, they also were asked to indicate their level of reward or pleasure on a 4-point Likert scale. Following monitoring, emphasis shifted to identifying values and goals within life areas that included family, social, and intimate relationships; education; employment and career; hobbies/recreation; volunteer work/charity; physical/health issues; spirituality; and anxiety-eliciting situations (Hayes, Strosahl, & Wilson, 1999). An activity hierarchy was then constructed in which 15 activities were rated from “easiest” to “most difficult” to accomplish. Using the master activity log and behavioral checkout to monitor progress, patients progressively moved through the hierarchy, from easier behaviors to the more difficult. The process of assigning behavioral activation goals began in Session 3. Weekly goals were recorded on a behavioral checkout form that the patient returned to therapy each week. At the start of each session, the behavioral checkout was examined and discussed, with the following weekly goals established as a function of patient success or difficulty. Although not outside the scope of BATD, attention to ongoing cancer treatment or cancer survivor issues were not directly addressed, in the former case because all breast cancer patients in this sample were provided psychotherapy following cancer treatment. In total, BATD involved eight sessions of approximately 1 hr in duration.

Therapists and Treatment Integrity

Advanced clinical psychology (doctoral) students served as therapists in this study. All therapists were skilled in the administration of BATD, had been trained by the principal investigator (DH), and had been regularly practicing BATD with patients for a minimum of 2 years prior to the study. To ensure competent provision of BATD, we audiotaped all sessions, and all therapists met for weekly individual supervision meetings with the principal investigator (DH). A total of 15% of tapes were selected randomly for ratings of therapist competence and adherence by an independent evaluator with expertise in behavioral therapy. Ratings were made on Likert scales ranging from 0 (no adherence/competence) to 8 (complete adherence/competence) on a session-by-session basis, with ratings based on adherence and ability in completing session objectives highlighted in the BATD treatment manual. Consistent with the very high ratings previously reported for the entire BATD sample (Hopko, Armento, et al., 2011), ratings of sessions conducted with this patient sample indicated high therapist adherence (M = 7.2; SD = 0.7) and competence (M = 6.9; SD = 1.0) to the BATD protocol.

Procedure

Following recruitment and screening procedures, eligible participants were administered the ADIS–IV and all self-report measures. All psychological assessments and treatment sessions were conducted at the Cancer Institute. Advanced doctoral students in clinical psychology conducted the comprehensive assessments. Patients subsequently engaged in their 8-week (one-on-one) treatment. At the beginning of each session, the BDI–II and EROS were completed to assess depression and environmental reward. For the purposes of this study, the master activity logs and behavioral checkouts were reviewed to assess the number of activities assigned and the proportion of activities completed by each patient.

Statistical Analyses

BA assignments commenced in the third session, meaning that the impact of the quantity and proportion of activities completed as they related to depression and environmental reward began to be assessed at Week 4 of treatment through the posttreatment assessment. Accordingly, the longitudinal data consisted of six observations: Sessions 4 (Data Point 1) through 8 and posttreatment (Data Point 6)]. Growth curve modeling was used to test levels of change in a dependent variable over time and incorporated within-subject and between-subjects predictors. The general model was as follows:  

The value of the dependent variable for patient i at time t was equal to a subject-specific intercept plus a subject-specific time slope. Setting t1 = 0, t2 = 1, t3 = 2, and so on; the intercept γ0i is the expected value for patient i at the first observation. If the expectation is that all patients have a similar baseline score that is not dependent on other variables, it can be written as a function of the overall average baseline across all individuals plus a random noise component representing person-specific differences from the mean.  

The value of γ1i reflects how rapidly each patient’s score on the dependent variable changes over time. Because each patient has her own growth trajectory, differences in the intercept and slopes are also modeled by individual-specific covariates. For example, it is predicted that depression severity declines more quickly as patients engage in more activities. Thus, we can model γ0i as a function of total activities completed (or proportion completed).   Substituting:   where the coefficient on the interaction tests the significance of total activities completed on time. Because there is an interaction, a main effect for total activities completed also is included. This is done by adding total activities completed to the model for the intercept.  

This yields the final model:  

The beta parameters represent fixed effects, or the average slopes and intercept across all patients in the sample. The r parameters represent random effects. They are not estimated as traditional regression coefficients. Rather, they are summarized by their variances as variance components. The larger the variance component, the greater the variability in growth trajectories between patients.

The previous model represents a between-subjects analysis, since it uses the total number of activities completed across time as the primary predictor. The data also contain week-specific values for the number of activities completed and proportion completed. Growth models also allow for the inclusion of time-varying predictors. In this case, the model simplifies to:  

Due to perfect multicollinearity between the time-varying activities measure and the total activities completed variable, the within-subject model was tested separately from the between-subjects model. Both were estimated to determine whether there were differences between week-specific activities completed and the total number of activities completed across the assessment period. The models described were all estimated using the MIXED command in SPSS Version 20.

Finally, it is also possible to test for mediation using a growth curve model, although it is more complicated for longitudinal relative to cross-sectional data (Selig & Preacher, 2009). Testing mediation is simplified in within-subject analyses, since the between-subjects model involves an interaction. The process for within-subject data amounts to fitting a simultaneous equations model in which the activities variable is both a predictor of depression severity and an outcome determined by environmental reward. Due to the fact that both equations involve random effects (as they are both growth models with time-varying variables), this part of the estimation was done using Mplus (Version 6.1; Muthén & Muthén, 2007). Standard errors for the indirect effects were estimated using bootstrapping.

The first hypothesis predicted that the number of activities assigned would increase as therapy progressed. For this hypothesis, the number of activities at time point t was the dependent variable in a growth model, and time was the sole predictor. The second hypothesis predicted that general compliance (proportion of activities completed) would increase as therapy progressed. This assertion was tested in the same manner as the first hypothesis, except that proportion of activities completed was the dependent variable. The third hypothesis predicted that depression severity would systematically decrease as the proportion of activities completed increased. In this case, depression was the dependent variable in the growth model. In the between-subjects model, the total proportion of activities completed was the predictor, and its interaction with time was tested to determine if a higher level of compliance resulted in quicker reductions in depression. A within-subject analysis determined if variations in weekly compliance had short-term effects on depression. A mediation analysis assessed whether any observed relationship between compliance and depression was partially or fully accounted for by the relationship between compliance and environmental reward. The third hypothesis also predicted that compliance (proportion of activities completed) would lead to increased environmental reward. Due to the simultaneous equations used to test mediation in Hypothesis 3, this test was integrated into the same model. Specifically, the model included an equation in which environmental reward was a direct consequence of compliance. If the coefficient for the effect of compliance on reward was significant, Hypothesis four would be supported. The fourth hypothesis was similar to the third, except total activities completed, rather than proportion completed, was the dependent variable. The same tests for direct effects and mediation were conducted. The fifth hypothesis predicted that the proportion of activities and quantity of completed activities would significantly impact treatment response and remission. Because each of these dependent variables was measured on a dichotomous scale, logistic regression was used. The independent variables were total proportion of activities completed and total quantity of activities completed. Results are reported as odds ratios, where a one-unit increase in the proportion of activities completed (or quantity of activities completed) is associated with a β increase in the odds of response or remission.

Response and Remission Criteria

Consistent with methods highlighted in previous trials of BA (Dimidjian et al., 2006; Hopko, Armento, et al., 2011), response represented significant symptomatic improvement, whereas remission represented improvement to the point of being asymptomatic within normal range. On the BDI–II, response was defined as at least a 50% reduction from baseline. Remission was defined as scores ≤ 10 on the BDI–II.

Supporting the first hypothesis and consistent with the progressive framework of BATD, results indicated that the number of assigned activities and number of completed activities significantly increased over time. As illustrated in Figure 1, during the first week of BA assignments (i.e., following Session 3), the average number of assigned activities was 12 (95% confidence interval, or CI, [9.29, 14.62]), nearly doubling to 23.39 (95% CI [17.84, 28.94]) at the conclusion of treatment. Additionally, there was greater variability in the number of assigned activities at the end of treatment. The number of activities assigned was expected to increase weekly by 2.16 (SE = 0.49, p < .01). However, the rate of increase was higher for some individuals and lower for others (σ² = 4.35, SE = 1.66, p = .01). There was also significant variability in the initial number of assigned activities (σ² = 29.36, SE = 14.21, p = .04). Patients who initially had fewer assigned activities had more substantial increases throughout treatment. The trend of completed activities over time is highlighted in Figure 2. During the first week of treatment, the average number of completed activities was 12.91 (95% CI [9.90, 15.93]) and 22.56 (95% CI [17.03, 28.10]) in the final week. Patients on average completed an additional 1.9 (SE = 0.51, p < .01) activities each week.

Figure 1. Average number of assigned behaviors. CI = confidence interval.

Figure 2. Average number of completed behaviors. CI = confidence interval.

When examining Hypothesis 2, we found that results were nonsignificant and indicated that the proportion of activities completed did not systematically increase during the course of BATD (B = −0.01, SE = 0.01, p = .36). As illustrated in Figure 3, following the first assignment, the average proportion of activities completed was 1.12 (95% CI [0.95, 1.30]), compared with 1.00 (95% CI [0.85, 1.16]) following the final activation session. The proportion of activities completed was essentially unchanged over time, primarily due to a ceiling effect whereby all patients were largely compliant with behavioral activation assignments.

Figure 3. Proportion of completed behaviors. CI = confidence interval.

Results partially supported Hypotheses 3 and 4. Figure 4 displays depression severity as a function of treatment session. Significant patient improvement was evident across behavioral activation sessions, whereby during the first week, the average BDI–II score was 14.83 (95% CI [11.53, 18.13]), decreasing to 10.04 (95% CI [6.87, 13.21]) by the final week of BATD. Figure 5 also shows a trend for increased environmental reward over time as a function of behavioral activation, with initial environmental reward of 26.13 (95% CI [24.45, 27.81]) increasing to 29.22 (95% CI [27.34, 31.06]) by the final week of treatment. Between-subjects analysis demonstrated that the average proportion of activities completed was significantly associated with decreased depression (B = −13.53, SE = 6.97, p < .05). The interaction between average proportion of completed activities and time was tested to determine if higher levels of compliance led to faster reductions in depression. The interaction was not significant (B = −0.12, SE = 1.11, p = .91). When a similar model was applied with environmental reward as the dependent variable, the average proportion of activities completed did not have a significant main effect (B = 2.19, SE = 3.69, p = .56) or interaction with time (B = 0.33, SE = 0.68, p = .63).

Figure 4. Average depression (Beck Depression Inventory–II; BDI–II) scores. CI = confidence interval.

Figure 5. Average Environmental Reward Observation Scale scores. CI = confidence interval.

Within-subject analysis incorporated week-specific values for proportion of completed activities. The results reinforce findings from the between-subjects model. As the proportion of activities completed in a given week increased, depression severity also decreased for that week (B = −4.04, SE = 1.40, p < .01). This result was observed beyond the general trend of diminishing depression severity, which was also significant (B = −1.24, SE = 0.27, p < .01). Thus, higher levels of compliance with behavioral activation assignments were effective in attenuating depression. As in the between-subjects model, the within-subject model showed that compliance did not have an effect on environmental reward. That is, the proportion of activities completed in a given week was not significantly related to that week’s score on the environmental reward scale (B = 0.94, SE = 0.81, p = .25), although the time trend was significant (B = 0.65, SE = 0.12, p < .01). With no evidence that proportion of completed activities had a significant effect on environmental reward, it was unlikely that the relationship between proportion of completed activities and depression was mediated by environmental reward. A multilevel mediation model confirmed there was no indirect effect (B = .026, SE = 0.037, p = .48).

When the total number of completed activities rather than proportion of completed activities was the primary predictor, quantity of completed activities did not affect depression severity either directly (B = −0.03, SE = 0.05, p = .53) or through its interaction with time trend (B = 0.00, SE = 0.01, p = .70). Only the time trend was significant in the model, with average depression severity decreasing each week (B = −1.48, SE = 0.78, p < .05). Similarly, findings showed a null effect of total completed activities on environmental reward. The main effect of total activities completed was not significant (B = 0.01, SE = 0.02, p = .71), and neither was the interaction with time (B = 0.00, SE = 0.00, p = .89). This was also demonstrated in the within-subject models, with total activities completed in a given week having no effect on that week’s depression score (B = 0.00, SE = 0.06, p = .94) and no effect on the respective week’s environmental reward score (B = −0.02, SE = 0.03, p = .59). With no effect of quantity of completed activities on depression severity or environmental reward, the key components of mediation were absent. Estimating a mediation model in Mplus yielded a nonsignificant within-subject indirect effect of −0.966 (SE = 1.128, p = .39), confirming the lack of mediation.

Logistic regression was used to examine Hypothesis 5, whether treatment response and remission were more likely to occur with a greater proportion of activities completed and greater quantity of activities completed. For treatment response, the average proportion of activities completed was significant (−21.19, SE = 9.33, p = .02). When patients completed all assigned activities, BDI–II treatment response was evident across all patients, exp (B) < .01, (95% CI [.00, .06]). The high pseudo R² values (Cox & Snell = .44, Nagelkerke = .64) also suggested very strong effect sizes. For depression remission, results were again significant (B = −9.97, SE = 4.95, p = .04). The odds ratio suggested that for those patients who completed all assigned activities, the odds of not achieving remission were essentially zero, exp (B) < .01 (95% CI [0.00, 0.77]). Model fit statistics indicated that the proportion of activities completed was a good predictor of remission outcomes (Cox & Snell = .27, Nagelkerke = .38). No effect was found when considering the predictive power of quantity of completed activities on BDI–II treatment response, B = −0.01, SE = 0.01, p = .55; exp (B) = .99 (95% CI [0.97, 1.02]), or remission, B = .00, SE = 0.01, p = .79; exp (B) = 1, (95% CI [0.98, 1.02]). Thus, relative to quantity of activities completed, treatment compliance was a better predictor of treatment outcome.

Contemporary BA treatments aim to attenuate depression via increased RCPR. Although specific intervention strategies differ across BA protocols, structured activation assignments are common to all approaches. This study examined longitudinal data to better explicate the process of change in BATD via growth curve modeling and examining relationships among the quantity of activities completed, proportion of activities completed (i.e., therapeutic compliance), environmental reward, and depression reduction. Findings demonstrate that while the average number of assigned and completed activities systematically increased over time, there was no progressive change in therapeutic compliance, with overall compliance being exceptionally high throughout psychotherapy. Extraordinary patient compliance with BATD may be reflective of a number of patient-centered, therapy-related, and social and economic factors, and potentially high therapist competence in assigning and reviewing homework (Jin, Sklar, Oh, & Li, 2008; Weck, Richtberg, Esch, Hofling, & Stangier, 2013). This finding is significant in that it supports the feasibility and tolerability of BATD for patients presenting with complex clinical presentations including a coexistent psychological disorder and medical illness.

When examining the effect of therapeutic compliance and quantity of completed activities on depression, study hypotheses were partially supported. Results indicated that from pre- to posttreatment, depression decreased as patients completed a higher proportion of activities. On a more microanalytical level, results revealed systematic reductions in depression during weeks where therapeutic compliance was highest. Somewhat unexpectedly, there was no significant effect on depression as a function of increased quantity of completed activities. Therefore, results suggest patient adherence to behavioral assignments is more critical in reducing depression relative to simply completing a greater number of activities. In fact, when examining the impact of therapeutic compliance on treatment response and remission, results were highly compelling and emphasize the importance of patient compliance toward positive BATD treatment outcome. Indeed, patient compliance with activation assignments resulted in favorable BATD treatment outcome. In terms of understanding BATD process of change, completing a greater proportion of assigned activities not only presumably increases RCPR but also likely facilitates greater self-efficacy, accomplishment, and mastery within valued life areas, and subsequent reductions in depression.

The finding of a limited relationship between quantity of activities completed and depression reduction is somewhat inconsistent with previous research showing a positive correlation between activity level and elevated mood. Accounting for this finding, it is conceivable that as the quantity of behavioral assignments increases, their significance pertaining to identified life values decreases. Indeed, initial behavioral assignments are not only prescribed according to their level of difficulty but also their relevance toward addressing the most important life values and their significance toward achieving the most principal life goals. Accordingly, a greater breadth of assigned behaviors may involve activities becoming more generic, less salient in terms of being value based and directly related to immediate life goals, and consequently less apt to result in environmental reinforcement and decreased depression. Another possible explanation may be related to the process of activity scheduling that involves assigning activities of progressively increasing difficulty (e.g., they require more time or effort, require underdeveloped skills, are associated with more anxiety, and avoidance motivation is stronger). With increased behavioral assignments, this process may involve inclusion of activities with less potential reinforcement value and possibly greater likelihood of aversive or unpleasant experiences and consequences that might prevent depression attenuation (e.g., lack of success, heightened anxiety). Highly important to acknowledge in this sample of breast cancer survivors, a committed engagement in fewer but highly valued (and possibly less difficult) activities might be the preferred mode of behavioral activation as patients seek to extract meaning in life, often following a very demanding breast cancer treatment regimen.

Analyses of the role of environmental reward revealed unexpected results given previous studies demonstrating the meditating effects of reinforcement on the relationship between behavioral activation and depression (Carvalho, Trent, et al., 2011; Carvalho & Hopko, 2011; Ryba & Hopko, 2012). Current findings yielded no support for either the proportion or quantity of completed activities as significantly related to self-reported environmental reward. However, study limitations should be taken into account when interpreting this finding, particularly the self-report method of assessing reinforcement. Because direct measurement of reinforcement would require direct observations of environmental contingencies across time and is not overly pragmatic, the practice of using self-report strategies to assess environmental reward, pleasure, and reinforcement is the common alternative. As such, environmental reward as measured in the current study may be an inadequate proxy for actual response-contingent positive reinforcement experienced in the natural environment.

Important to highlight, BA and increased environmental reinforcement likely is not the only critical mediator of change in BATD. For example, the sudden gain literature suggests the beneficial effects of BATD are at least partially independent of the activation process itself, with 50% of sudden gains, or large symptom improvements between one treatment interval, occurring prior to the activation process that commences in Session 3 (Hopko, Robertson, & Carvalho, 2009; Hunnicutt-Ferguson, Hoxha, & Gollan, 2012; Kelly, Cyranowski, & Frank, 2007; Tang & DeRubeis, 1999). This means that sudden gains may be partially reflective of developing therapeutic alliance, psychoeducation, environmental modification (i.e., reducing reinforcement for depressed behavior), and structured value assessment that occur in the first two BA sessions. Additionally, it is conceivable that sudden gains partially reflect self-activation in the absence of therapist guidance. In a recent study of sudden gains in depressed patients receiving BA (Hunnicutt-Ferguson et al., 2012) and consistent with previous data (Hopko et al., 2009), 67% of sudden gains occurred in the first few sessions of behavioral activation. In the present study, although increased proportion of activation assignments completed directly mediated depression reduction, it also is true that BDI–II depression severity had already decreased 45% from baseline (i.e., from moderate to mild depression) prior to beginning structured activation in Session 3. This early and significant reduction in depression symptoms prior to activation emphasizes the need to further examine nonspecific therapy factors such as patient motivation for treatment, perceived support, previous therapy experiences, or patient-specific risk/protective factors such as chronicity of depression and comorbid diagnoses (Hunnicutt-Ferguson et al., 2012). The important point here is that in the context of methodological problems with directly measuring environmental reinforcement and frequently large reductions in depression that occur prior to activation, environmental reinforcement may remain highly operative as a mediator in BATD—although other important mediators of change most definitely must be considered.

Although findings are provocative, several important study limitations are noteworthy. First, a larger sample size would have increased power and confidence in study findings. For example, although mediation models have been examined in smaller samples, a larger sample size would have allowed for better assessment of the potential mediating effects of environmental reward (Fritz & MacKinnon, 2007). Second, as this study followed from a randomized clinical trial and was not specifically designed to account for all possible mediators of change, future work should assess the impact of common factors on treatment outcome (e.g., therapeutic alliance, level of therapist reinforcement, patient self-efficacy). Third, it is conceivable that study recruitment strategies yielded a unique cohort of breast cancer patients that on some unmeasured variable may have been distinct from the population of breast cancer patients. Fourth, as discussed, the ideal method of assessing reinforcement via direct observations of environmental contingencies was not feasible, resulting in the use of a self-report measure of environmental reward (EROS). As “environmental reward” is not synonymous with “environmental reinforcement,” no definitive conclusion can be drawn about the relationship of reinforcement with treatment response to BATD. In addition, as the EROS was designed to assess environmental reward within “the past several months” but was administered weekly in this study, this measure characteristic could have complicated data interpretation and possibly contributed to Type II error in examining environmental reward as a mediating variable. Although still not optimal, the Reward Probability Index (RPI; Carvalho, Gawrysiak, et al., 2011) may have been a preferred proxy measure of environmental reinforcement as unlike the EROS, the RPI better assesses the construct of response-contingent positive reinforcement (i.e., number of reinforcers, availability of reinforcers, ability to obtain reinforcement, exposure to aversive events; Lewinsohn, 1974). Fifth, although BATD practitioners in this study had been trained by the senior author and had been educated in the principles and practice of BATD for a minimum of 2 years, they certainly would not be qualified as experts given their status as unlicensed clinical practitioners.

Sixth, daily diary logs were used to track activity assignments and completion. Although patients received careful instruction and reported procedural adherence, it cannot be definitively stated that activities were logged reliably or accurately. Seventh, because only 55% of the original sample of patients treated with BA (Hopko, Armento, et al., 2011) returned all behavioral monitoring logs, this analyzable subsample may not be entirely representative of the entire BA cohort or breast cancer patients in general. Although this subsample and holdout sample (i.e., those not returning monitoring logs) did not statistically differ on all primary psychological, demographic, and cancer-related study variables, had comparable levels of pretreatment depression severity, and had similar response and remission rates to BATD, it is conceivable that between-group differences existed on some unmeasured variable(s). As many anecdotal therapist reports and audiotaped session transcripts indicated, however, a majority of (holdout) BATD patients who did not successfully return behavioral monitoring logs to therapy sessions nevertheless communicated strong adherence to weekly activation assignments. Nonetheless, significant findings cannot be generalized to the holdout sample or population of breast cancer patients with absolute confidence. Eighth, the study examined the quantity and proportion of completed activities but did not differentiate among types of activities (e.g., social, physical, educational). Categorization of activities may have provided a more detailed understanding of whether engagement in certain types of activities was relevant in the process and outcome of BA. Finally, since the study examined process of change among a highly educated and largely White sample of depressed breast cancer patients, further inquiry into the generalizability of study findings to other patient samples is warranted.

In closing, study findings provide novel insight into the process and outcome of BA and have important clinical and research implications. Results highlight the efficacy of BA and suggest therapeutic compliance is a vital component toward increasing the probability of positive treatment outcome. Highly provocative, findings also suggested that “more” behavioral activation as defined by an increased quantity of completed behaviors does not necessarily correspond with improved treatment response. Further research should more systematically examine dose-response relationships associated with BA while also being mindful of whether certain categories or types of activities are more relevant toward conceptualizing treatment outcome. Given the empirical support and practicality of BA, continued investigation of process factors is warranted. Dismantling studies of treatment components of differing BA approaches also may be beneficial toward better isolating strategies most essential to engendering healthy behaviors. Although in the present study medication use was not examined in primary analyses (but used as a covariate) due to findings that medication use was unassociated with treatment response and remission in this sample (Hopko, Armento, et al., 2011), because combined therapy may be associated with improved treatment response in some depressed patients (Hollon et al., 2005), this issue also should be further examined in understanding the process and outcome of BA. In addition, although factors associated with treatment failure in behavior activation have been discussed that include noncompliance with behavioral assignments (Hopko, Magidson, & Lejuez, 2011), strategies most effective in promoting compliance with activation are largely unknown and require further investigation. The uncomplicated and easily disseminated approach of BA has many potential applications in a broad range of clinical settings and by a diverse network of providers (Ekers, Richards, McMillan, Bland, & Gilbody, 2011). The more precisely the process of change is understood, the better equipped researchers and clinicians will be to further refine and deliver the most parsimonious and efficacious form of BA.

¹ As reported, the study and holdout samples did not significantly differ on any primary psychological, demographic, and cancer-related variables. Although sophisticated statistical procedures such as multiple imputation for missing data were considered, because of excessive missing data (64%) in the holdout sample on the behavioral monitoring logs on which patients were to record compliance with assigned activities, multiple imputation procedures were not used as doing so would have violated statistical assumptions (Little & Rubin, 2002; Rubin, 1987) and potentially rendered data invalid and non-interpretable.

² All growth curve modeling and mediation analyses also were conducted using medication status (i.e., not medicated, stabilized on medication) and pretreatment depression severity as covariates. Results remained consistent with those reported.

Addis, M. E., & Martell, C. R. (2004). Overcoming depression one step at a time: The new behavioral activation approach to getting your life back. New York, NY: New Harbinger.

Armento, M. E. A., & Hopko, D. R. (2007). The development and validation of the Environmental Reward Observation Scale (EROS). Behavior Therapy, 38, 107–119. doi:10.1016/j.beth.2006.05.003

Arnau, R. C., Meagher, M. W., Norris, M. P., & Bramson, R. (2001). Psychometric evaluation of the Beck Depression Inventory–II with primary care patients. Health Psychology, 20, 112–119. doi:10.1037/0278-6133.20.2.112

Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for Beck Depression Inventory–II. San Antonio, TX: Psychological Corporation.

Brown, T. A., Di Nardo, P., & Barlow, D. H. (1994). Anxiety Disorders Interview Schedule for DSM–IV. San Antonio, TX: The Psychological Corporation.

Carvalho, J. P., Gawrysiak, M. J., Hellmuth, J. C., McNulty, J. K., Magidson, J. F., Lejuez, C. W., & Hopko, D. R. (2011). The Reward Probability Index: Design and validation of a scale measuring access to environmental reward. Behavior Therapy, 42, 249–262. doi:10.1016/j.beth.2010.05.004

Carvalho, J. P., & Hopko, D. R. (2011). Behavioral theory of depression: Reinforcement as a mediating variable between avoidance and depression. Journal of Behavior Therapy and Experimental Psychiatry, 42, 154–162. doi:10.1016/j.jbtep.2010.10.001

Carvalho, J. P., Trent, L. R., & Hopko, D. R. (2011). The impact of decreased environmental reward in predicting depression severity: Support for behavioral theories of depression. Psychopathology, 44, 242–252. doi:10.1159/000322799

Cuijpers, P., van Straten, A., & Warmerdam, L. (2007). Behavioral activation treatments of depression: A meta-analysis. Clinical Psychology Review, 27, 318–326. doi:10.1016/j.cpr.2006.11.001

Dimidjian, S., Barrera, M., Martell, C., Munoz, R. F., & Lewinsohn, P. M. (2011). The origins and current status of behavioral activation treatments for depression. Annual Review of Clinical Psychology, 7, 1–38. doi:10.1146/annurev-clinpsy-032210-104535

Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., . . .Jacobson, N. S. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74, 658. doi:10.1037/0022-006X.74.4.658

Dobson, K. S., Hollon, S. D., Dimidjian, S., Schmaling, K. B., Kohlenberg, R. J., Gallop, R. J., . . .Jacobson, N. S. (2008). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the prevention of relapse and recurrence in major depression. Journal of Consulting and Clinical Psychology, 76, 468–477. doi:10.1037/0022-006X.76.3.468

Ekers, D., Richards, D., & Gilbody, S. (2008). A meta-analysis of randomized trials of behavioural treatment of depression. Psychological Medicine: A Journal of Research in Psychiatry and the Allied Sciences, 38, 611–623.

Ekers, D., Richards, D., McMillan, D., Bland, J. M., & Gilbody, S. (2011). Behavioral activation delivered by the non-specialist: Phase II randomized controlled trial. The British Journal of Psychiatry, 198, 66–72. doi:10.1192/bjp.bp.110.079111

Ferster, C. B. (1973). A functional analysis of depression. American Psychologist, 28, 857–870. doi:10.1037/h0035605

Fritz, M. S., & MacKinnon, D. P. (2007). Required sample size to detect the mediated effect. Psychological Science, 18, 233–239. doi:10.1111/j.1467-9280.2007.01882.x

Grosscup, S. J., & Lewinsohn, P. M. (1980). Unpleasant and pleasant events and mood. Journal of Clinical Psychology, 36, 252–259. doi:10.1002/1097-4679(198001)36:1<252::AID-JCLP2270360131>3.0.CO;2-S

Hamilton, M. (1960). A rating scale for depression. Neurology, Neurosurgery and Psychiatry, 23, 56–61.

Hayes, S. C., Strosahl, K. D., & Wilson, K. G. (1999). Acceptance and commitment therapy: An experiential approach to behavior change. New York, NY: Guilford Press.

Hollon, S. D., Jarrett, R. B., Nierenberg, A. A., Thase, M. E., Trivedi, M., & Rush, A. J. (2005). Psychotherapy and medication of adult and geriatric depression: Which monotherapy or combined treatment?Journal of Clinical Psychiatry, 66, 455–468. doi:10.4088/JCP.v66n0408

Hopko, D. R., Armento, M. E. A., Cantu, M. S., Chambers, L. L., & Lejuez, C. W. (2003). The use of daily diaries to assess the relations among mood state, overt behavior, and reward value of activities. Behaviour Research and Therapy, 41, 1137–1148. doi:10.1016/S0005-7967(03)00017-2

Hopko, D. R., Armento, M. E. A., Robertson, S., Ryba, M. M., Carvalho, J. P., Colman, L. K., . . .Lejuez, C. W. (2011). Brief behavioral activation and problem-solving therapy for depressed breast cancer patients: Randomized trial. Journal of Consulting and Clinical Psychology, 79, 834–849. doi:10.1037/a0025450

Hopko, D. R., & Lejuez, C. W. (2007). A cancer patient’s guide to overcoming depression and anxiety: Getting through treatment and getting back to your life. New York, NY: New Harbinger.

Hopko, D. R., Lejuez, C. W., Ruggiero, K. J., & Eifert, G. H. (2003). Contemporary behavioral activation treatments for depression: Procedures, principles, and progress. Clinical Psychology Review, 23, 699–717. doi:10.1016/S0272-7358(03)00070-9

Hopko, D. R., Magidson, J. F., & Lejuez, C. W. (2011). Treatment failure in behavior therapy: Focus on behavioral activation for depressed cancer patients. Journal of Clinical Psychology, 67, 1106–1116. doi:10.1002/jclp.20840

Hopko, D. R., & Mullane, C. M. (2008). Exploring the relation of depression and overt behavior with daily diaries. Behaviour Research and Therapy, 46, 1085–1089. doi:10.1016/j.brat.2008.05.002

Hopko, D. R., Robertson, S., & Carvalho, J. (2009). Sudden gains in depressed cancer patients treated with behavioral activation therapy. Behavior Therapy, 40, 346–356. doi:10.1016/j.beth.2008.09.001

Hopko, D. R., Ryba, M. M., McIndoo, C., & File, A. (in press). Behavioral activation. In A. M.Nezu & C. M.Nezu (Eds.), The Oxford handbook of cognitive and behavioral therapies. Oxford, United Kingdom: Oxford University Press.

Hunnicutt-Ferguson, K., Hoxha, D., & Gollan, J. (2012). Exploring sudden gains in behavioral activation treatment for major depressive disorder. Behaviour Research and Therapy, 50, 223–230. doi:10.1016/j.brat.2012.01.005

Jin, J., Sklar, G. E., Oh, V. M., & Li, S. C. (2008). Factors affecting therapeutic compliance: A review from the patient’s perspective. Therapeutics and Clinical Risk Management, 4, 269–286.

Kanter, J. W., Manos, R. C., Bowe, W. M., Baruch, D. E., Busch, A. M., & Rusch, L. C. (2010). What is behavioral activation? A review of the empirical literature. Clinical Psychology Review, 30, 608–620. doi:10.1016/j.cpr.2010.04.001

Katz, M. R., Kopek, N., Waldron, J., Devins, G. M., & Thomlinson, G. (2004). Screening for depression in head and neck cancer. Psycho-Oncology, 13, 269–280. doi:10.1002/pon.734

Kelly, M. A. R., Cyranowski, J. M., & Frank, E. (2007). Sudden gains in interpersonal psychotherapy for depression. Behaviour Research and Therapy, 45, 2563–2572. doi:10.1016/j.brat.2007.07.007

Lejuez, C. W., Hopko, D. R., Acierno, R., Daughters, S. B., & Pagoto, S. L. (2011). Ten year revision of the brief behavioral activation treatment for depression: Revised treatment manual. Behavior Modification, 35, 111–161. doi:10.1177/0145445510390929

Lejuez, C. W., Hopko, D. R., & Hopko, S. D. (2001). The brief behavioral activation treatment for depression (BATD): A comprehensive patient guide. Boston, MA: Pearson Custom.

Lewinsohn, P. M. (1974). A behavioral approach to depression. In R. M.Friedmann & M. M.Katz (Eds.), The psychology of depression: Contemporary theory and research (pp. 157–185). New York, NY: Wiley.

Lewinsohn, P. M., & Libet, J. (1972). Pleasant events, activity schedules, and depressions. Journal of Abnormal Psychology, 79, 291–295. doi:10.1037/h0033207

Lewinsohn, P. M., & Shaffer, M. (1971). Use of home observations as an integral part of the treatment of depression: Preliminary report and case studies. Journal of Consulting and Clinical Psychology, 37, 87–94. doi:10.1037/h0031297

Lewinsohn, P. M., & Shaw, D. A. (1969). Feedback about interpersonal behavior change: A case study in the treatment of depression. Psychotherapy and Psychosomatics, 17, 82–88. doi:10.1159/000286014

Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). New York, NY: Wiley.

MacPhillamy, D. J., & Lewinsohn, P. M. (1974). Depression as a function of levels of desired and obtained pleasure. Journal of Abnormal Psychology, 83, 651–657. doi:10.1037/h0037467

Martell, C. R., Addis, M. E., & Jacobson, N. S. (2001). Depression in context: Strategies for guided action. New York, NY: Norton.

Mazzucchelli, T., Kane, R., & Rees, C. (2009). Behavioral activation treatments for depression in adults: A meta-analysis and review. Clinical Psychology: Science and Practice, 16, 383–411. doi:10.1111/j.1468-2850.2009.01178.x

Muthén, L. K., & Muthén, B. O. (2007). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén.

Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley. doi:10.1002/9780470316696

Ryba, M. M., & Hopko, D. R. (2012). Gender differences in depression: Mediating effects of overt behavior and environmental reward as assessed through daily diary monitoring. Depression Research and Treatment, 12, 1–9.

Selig, J. P., & Preacher, K. J. (2009). Mediation models for longitudinal data in developmental research. Research in Human Development, 6, 144–164. doi:10.1080/15427600902911247

Sturmey, P. (2009). Behavioral activation is an evidence-based treatment for depression. Behavior Modification, 33, 818–829. doi:10.1177/0145445509350094

Tang, T. Z., & DeRubeis, R. J. (1999). Sudden gains and critical sessions in cognitive-behavioral therapy for depression. Journal of Consulting and Clinical Psychology, 67, 894–904. doi:10.1037/0022-006X.67.6.894

Weck, F., Richtberg, S., Esch, S., Hofling, V., & Stangier, U. (2013). The relationship between therapist competence and homework compliance in maintenance cognitive therapy for recurrent depression: Secondary analysis of a randomized trial. Behavior Therapy, 44, 162–172. doi:10.1016/j.beth.2012.09.004

Submitted: June 6, 2013 Revised: October 31, 2013 Accepted: November 12, 2013

Case Complexity as a Guide for Psychological Treatment Selection

Acknowledgement: We thank Jan R. Böhnke for helpful comments on an earlier version of this article.

A commonly held view in clinical psychology is that complex cases require suitably intensive interventions guided by formulations that account for obstacles to improvement (Tarrier, 2006). Clinical wisdom reflected in treatment textbooks suggests that a variety of factors can complicate treatment, such as chronic symptoms, comorbidity, personality disorders, physical illnesses, and so forth (Beck, 1998; Hawton, Salkovskis, Kirk, & Clark, 1989; Tarrier, Wells, & Haddock, 1998). Along these lines, Ruscio and Holohan (2006) proposed a list of more than 40 factors that characterize complex cases, clustered around several themes including symptoms, safety, physical, intellectual, personality, and other features. Evidently, case complexity is a heterogeneous concept and there is little consensus about the features that define such cases.

Moreover, the empirical literature casts doubt over the predictive value of many variables presumed to hinder the effectiveness of therapy (Garfield, 1994). A case in point is found in the study by Myhr et al. (2007), in which only five out of 10 variables thought to be indicative of poor suitability for cognitive therapy were (weakly) correlated with post-treatment outcomes. It is also well documented that clinicians’ prognostic assessment of patients tends to be inaccurate (Ægisdóttir et al., 2006; Grove & Meehl, 1996), often failing to identify cases at risk of poor treatment outcomes (Hannan et al., 2005). In another study, patients randomly assigned to brief manualized interventions offered in a stepped care model had comparable outcomes to patients whose treatments were selected and informed by clinical judgment (Van Straten, Tiemens, Hakkaart, Nolen, & Donker, 2006). Such evidence calls into question clinicians’ ability to match patients to treatments and supports current guidelines to apply a stepped care approach (National Institute for Health and Care Excellence [NICE], 2011). Overall, three key problems are apparent: a lack of conceptual clarity about complex cases, a gap between clinical judgment and research evidence, and limitations in clinicians’ ability to identify and select optimal treatments for complex cases.

Concerns regarding complexity are not exclusive to the practice of psychotherapy. The simultaneous growth and ageing of the general population have confronted many other areas of health care with the challenges of treating patients who present with multiple chronic conditions (Smith & O’Dowd, 2007), leading some to question the usefulness of evidence-based guidelines that are formulated for “prototypical” patients (Boyd et al., 2005; Tinetti, Bogardus, & Agostini, 2004). Consequently, theoretical models to account for case complexity in medicine have been proposed in the last decade. Some of these models conceptualize complexity as arising from a combination of clinical (e.g., diagnostic), biological, socioeconomic, cultural, environmental, and behavioral factors that are statistically associated with clinical prognosis (Safford, Allison, & Kiefe, 2007; Schaink et al., 2012). Individual patients may have protective or risk factors across these domains, and their overall complexity level results from the sum of risks. In an attempt to move beyond mere description, Shippee, Shah, May, Mair, and Montori (2012) proposed a cumulative complexity model that attempts to explain how risk factors accumulate and interact to influence health care outcomes. They proposed that (clinical, socioeconomic, cultural) risk factors complicate health care outcomes by disrupting the balance between patient workload (i.e., number and difficulty of daily life demands including self-care) and patient capacity (i.e., resources and limitations affecting ability to meet demands). From this perspective, effective health care for complex cases would not only require intensive treatment of acute symptoms and specific disease mechanisms, but also attending to wider biopsychosocial aspects that may redress the balance between demands and capacity. Common to these models are the focus on empirically supported prognostic factors, the consideration of factors across multiple domains, and the conceptual understanding of case complexity as resulting from the accumulation of risks and challenges to self-management.

Informed by these theoretical models emerging from the biomedical sciences, this study investigated the impact of case complexity in routine psychological care. Considering the problems outlined above, we sought to assess the merits of an actuarial, data-driven, cumulative model of defining case complexity. Specific objectives were (a) to identify prognostic variables associated with psychological treatment outcomes, (b) to develop an algorithm that could aid clinicians in identifying complex cases at risk of poor outcomes, (c) to determine whether or not complex cases respond differentially to treatments of differing levels of intensity, (d) to ascertain the extent to which patients are adequately matched to available stepped care interventions.

Setting and Interventions

This study was based on the analysis of clinical data routinely collected by a primary care psychological therapy service in Northern England. The study was approved as a service evaluation by the local National Health Service (NHS) Trust, which did not require formal ethical approval. The service offered low- and high-intensity interventions for depression and anxiety problems, as part of the Improving Access to Psychological Therapies (IAPT) program (Clark et al., 2009). Low-intensity treatments (LIT) consisted of brief (<8 sessions lasting 30 min) psychoeducational interventions based on principles of cognitive behavioral therapy (CBT). These were highly structured interventions, supported by didactic materials and delivered by a workforce of psychological wellbeing practitioners trained to a standard national curriculum (Bennett-Levy et al., 2010). High-intensity treatments (HIT) were lengthier (up to 20 sessions lasting around 60 min) interventions including CBT and counseling for depression. These interventions were also protocol-driven, delivered by postgraduate level counselors and psychotherapists, following national treatment guidelines (NICE, 2010) and competency frameworks (e.g., Roth & Pilling, 2008). All therapists practiced under regular clinical supervision (weekly or fortnightly) to ensure ethical practice and treatment fidelity. These interventions were organized in a stepped care model (NICE, 2011), where most patients initially accessed LIT and those with persistent and/or severe symptoms accessed HIT. Initial treatment assignment was determined by therapists who carried out standardized intake assessments.

Measures and Data Sources

Primary outcome measures

Patients accessing IAPT services self-complete standardized outcome measures on a session-to-session basis to monitor response to treatment. The Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001) is a nine-item screening tool for major depression, where each item is rated on a 0 to 3 Likert scale, yielding a total depression severity score between 0 and 27. A cut-off ≥10 has been recommended to detect clinically significant depression symptoms (Kroenke et al., 2001), and a difference of ≥6 points between assessments is indicative of reliable change (Richards & Borglin, 2011).

The Generalized Anxiety Disorder-7 (GAD-7; Spitzer, Kroenke, Williams, & Löwe, 2006) is a seven-item measure developed to screen for anxiety disorders. It is also rated using Likert scales, yielding a total anxiety severity score between 0 and 21. A cut-off score ≥8 is recommended to identify the likely presence of a diagnosable anxiety disorder (Kroenke, Spitzer, Williams, Monahan, & Löwe, 2007), and a difference of ≥5 points is indicative of reliable change (Richards & Borglin, 2011). Pre-treatment and last observed PHQ-9 and GAD-7 scores were available for analysis.

Other measures

The Work and Social Adjustment Scale (WSAS; Mundt, Marks, Shear, & Greist, 2002) is a measure of functioning across five domains: work, home management, social leisure activities, private leisure activities, family and close relationships. Each item is rated on a scale of 0 (no impairment) to 8 (very severe impairment), rendering a total functional impairment score between 0 and 40.

The Standardized Assessment of Personality–Abbreviated Scale (SAPAS) is an eight-item questionnaire developed to screen for the likely presence of a personality disorder (Moran et al., 2003). Each question prompts respondents to endorse specific personality traits (yes/no), yielding a total score between 0 and 8 where a cut-off >3 is indicative of cases with a high probability of diagnosable personality disorders. The WSAS and SAPAS were gathered at the time of initial assessments.

De-identified treatment and demographic data were also available, including information on referral sources, the intensity and sequence of treatments received (LIT and/or HIT along the stepped care pathway), age, gender, ethnicity and employment status. Formal diagnostic assessments were not carried out in routine care, but primary presenting problems noted in clinical records were available in summary form as group-level percentages.

Sample Characteristics

The study included case records for a total of 2,202 patients who had been discharged from the service at the time of data collection. Complete data (described earlier) were available for 1,512 (68.7%) cases. More than half were females (63.9%), with a mean age of 41.99 (SD = 14.54; range = 16 – 87) and of white British ethnic background (88.2%). A quarter (24.9%) of all cases were unemployed and/or in receipt of incapacity benefits. Approximately 59.9% were referred to treatment by general medical practitioners; the remainder self-referred (24.3%) or were referred by other social and health care providers (15.8%). The presenting problems noted in clinical records were depression (21.0%), recurrent depression (6.6%), obsessive–compulsive disorder (4.4%), adjustment disorders (5.7%), somatoform disorders (0.4%), eating disorders (0.4%), phobic disorders (5.7%), other anxiety disorders (42.4%), and unspecified mental health problems (13.4%). Mean baseline severity scores for the whole cohort were PHQ-9 = 14.86 (SD = 6.33), GAD-7 = 13.27 (SD = 5.07), WSAS = 18.39 (SD = 9.46), SAPAS = 3.82 (SD = 1.89; cases with SAPAS >3 = 54.2%). Many patients had comorbid presentations, where 71.4% of cases had case-level symptoms in both PHQ-9 and GAD-7. Approximately 76.6% of patients were initially assigned to LIT and 23.4% were initially assigned to HIT. Overall, 40.6% only accessed LIT, 36.0% accessed LIT + HIT, and 23.4% only accessed HIT. Overall, 31.3% dropped out of treatment (32.2% of those initially assigned to LIT; 28.5% of those initially assigned to HIT).

Statistical Analysis

Consistent with the objectives of the study, data analyses were performed in four stages aiming to develop, validate and assess the clinical utility of a cumulative complexity model. The primary analyses were carried out in the dataset of cases with complete data (N = 1,512). Following a cross-validation approach, we partitioned this dataset into two random halves which were treated as estimation (N = 755) and validation (N = 757) samples. In order to assess the potential influence of missing data, a single imputed estimation sample (N = 1,108) was derived using an expectation-maximization method (Schafer & Olsen, 1998) and was used for sensitivity analyses described below.

Stage I involved the development of a prognostic index and classification method to identify complex cases in routine care. The dependent variable in all models was a binary indicator of post-treatment reliable and clinically significant improvement (RCSI), with separate models for depression (PHQ-9) and anxiety (GAD-7) measures. RCSI was determined using the criteria proposed by Jacobson and Truax (1991), based on combining reliable change indices for PHQ-9 (≥6) and GAD-7 (≥5) described by Richards and Borglin (2011) and diagnostic cut-offs for each measure (PHQ-9 < 10; GAD-7 < 8). The dependent variable was coded as follows: 0 = RCSI; 1 = no RCSI, such that the prognostic models would be constructed to identify (more complex) cases with increased probability of poor outcomes.

As an initial variable screening step, we used univariate logistic regressions to examine the goodness-of-fit (based on −2 log likelihood test and magnitude of AIC and BIC statistics) of linear and nonlinear trends for continuous variables, as well as alternative ways to model the SAPAS Questionnaire (as a total score, dichotomized based on a cut-off >3, or entered as a series of eight binary items). Entering all eight SAPAS binary items yielded the best fitting models in preliminary tests (i.e., lowest AIC and BIC, significant −2 log likelihood tests) and confirmed that only five items were significant (p < .05) predictors of outcome. Furthermore, baseline severity (PHQ-9, GAD-7), impairment (WSAS) and age variables were optimally modeled using nonlinear trends. Age was rescaled to ordinal decade groups (e.g., teens, twenties, thirties, etc.) and reverse scored (oldest group coded 0, youngest group coded 6) on the basis of the observed trend of correlations between age and RCSI.

Informed by these preliminary tests, we applied penalized categorical regressions with optimal scaling (CATREG-Lasso) in the main analysis. CATREG applies classical linear regression to predictor variables that are transformed to categorical quantifications which are optimally suited to explore nonlinear relations in the data (Gifi, 1990). Continuous variables were thus transformed using a monotonic spline scaling level to examine nonlinear associations with the dependent variable. Variable selection and regularization were performed combining the Lasso procedure (Tibshirani, 1996) and the .632 bootstrap resampling method (Efron & Tibshirani, 1997). The Lasso imposes a penalty term that shrinks coefficients toward zero, penalizing the sum of the squared regression coefficients. This yields more generalizable prediction equations compared with conventional regression models which are prone to overfitting and are less reliable in the presence of multicollinearity. Because using different penalty terms results in different shrunken coefficients, resampling techniques are often used to determine an optimal penalty. The .632 bootstrap resampling method is a smoothed version of the leave-one-out cross-validation strategy, which permits the estimation of a model’s expected prediction error. This resampling method was applied 1,000 times to each Lasso model, iteratively increasing the penalty term in 0.01 units, until all coefficients were shrunk to zero. The one-standard-error rule was applied to select the most parsimonious Lasso model within one standard error of the model with minimum expected prediction error. The predictors entered into CATREG-Lasso models included clinical (baseline PHQ-9, GAD-7, WSAS), personality (SAPAS items 1, 2, 3, 5, 7) and demographic variables (age groups, gender, ethnicity, employment status). Shrunken coefficients from the optimal models were used to calculate a prognostic index (PI) for each patient, where a higher PI denotes poorer prognosis. PIs were retained in the CATREG quantifications scale, with signed and continuous scores centered at zero.

The above procedure was conducted in the estimation samples with complete and imputed data, allowing us to compare the area under the curve (AUC) for the PIs derived from each dataset as an indicator of predictive accuracy. PIs derived using complete and imputed samples had comparable AUC statistics albeit with some shrinkage observed in the imputed dataset (PHQ-9: 0.67 ± 0.04 vs. 0.63 ± 0.05; GAD-7: 0.74 ± 0.04 vs. 0.66 ± 0.04). Therefore, subsequent analyses were applied in the dataset with complete data.

In Stage II, we applied receiver operating characteristic (ROC) curve analysis (Altman & Bland, 1994) in the estimation sample to determine empirical cut-offs that optimally balanced sensitivity and specificity on each PI. Consistent with our assumptions about clinical complexity, cases where both (PHQ-9 and GAD-7) PIs were above empirical cut-offs were classed as complex (Cx), and others (including all those with subclinical symptoms) were classed as standard (St) cases. The agreement of both PIs was taken as a conservative means of minimizing “false positive” classifications, and limiting the Cx classification to cases with the poorest prognoses across both outcome domains. We then tested our assumptions about prognosis and cumulative complexity in the validation subsample, with cases whose symptoms were above diagnostic cut-offs for each outcome measure (PHQ-9: N = 675; GAD-7: N = 755). ROC curve analyses were used to assess how well the PIs (using Lasso-based shrunken coefficients from the estimation sample) performed out-of-sample (in a statistically independent validation sample). In addition, separate logistic regression models were applied for each outcome (PHQ-9, GAD-7), where the dependent variable was post-treatment RCSI status (0 = no RCSI; 1 = RCSI) and the predictors included case complexity (0 = Cx, 1 = St) controlling for baseline severity of symptoms (PHQ-9 or GAD-7, respectively).

Stage III analyses were also conducted in the validation sample. A logistic regression model predicting (HIT vs. LIT) group membership based on clinical and demographic characteristics was performed to estimate propensity scores, denoting the predicted probability of completing a treatment episode at HIT. Propensity scores were entered as a covariate in subsequent analyses to control for confounding by indication. Next, logistic regression models were applied with RCSI status as a dependent variable, entering baseline severity (PHQ-9 or GAD-7, respectively), propensity scores, and treatment pathway (LIT or HIT only vs. LIT + HIT) as predictors. The models were performed separately in the subgroups of Cx (N = 269) and St (N = 425) cases (with available data to estimate propensity scores), to minimize multicollinearity between propensity scores and case complexity dummy variables in the same model.

In Stage IV, we assessed the extent to which initial treatment assignment (LIT or HIT) determined by clinical judgment was consistent with the assignment that would be indicated by the prognostic method described previously. A prognostic treatment assignment was coded for all patients, where starting at HIT was recommended for Cx cases and starting at LIT was recommended for all other cases. Next, agreement codes were noted for each case in the full sample, where “1” indicated agreement between clinical judgment and prognosis, and “0” indicated disagreement. Agreement codes were aggregated across the entire sample to estimate a “hit rate,” denoting the percentage of cases where clinicians’ decisions converged with a prognostic strategy for treatment assignment. Next, we applied Cohen’s kappa across agreement codes to derive a Treatment Matching precision (TMaP) score, which takes into account the probability that “hit rates” may be due to chance. The TMap score is therefore a robust measure of convergence between clinical and empirical decision-making strategies, ranging between 1 (perfect agreement) and −1 (complete disagreement), where 0 is indicative of agreement by chance. TMaP scores were estimated for the full sample and for individual clinicians that undertook initial assessments and made decisions about treatment assignment for at least 20 patients (to eschew extreme scores in caseloads with small base rates).

Estimation of Prognostic Equations

Using the CATREG-Lasso procedure in the estimation sample, we arrived at prognostic models that explained between 9% (PHQ-9: optimal scaling adjusted R² = 0.09) and 15% (GAD-7: adjusted R² = 0.15) of variance in posttreatment RCSI. Regression and ROC curve model estimates for each outcome measure are presented in Table 1 (with detailed outputs in Appendix A). Several predictors were selected into optimal Lasso models, including demographic (age, ethnicity, employment), personality (SAPAS items: 2 = interpersonally avoidant, 3 = suspicious, 5 = impulsive, 7 = dependent), and clinical features (baseline PHQ-9, GAD-7, WSAS).

Estimation of Prognostic Indices Using Penalized Categorical Regression With Optimal Scaling

The R² share statistic reflects the relative contribution of each predictor to the model’s overall adjusted R², after partialing out the specific and combined effects of the other variables. In the depression model, demographics had relatively greater explanatory influence (22.5%) relative to personality (14.7%) and clinical features (15%), although the remaining R² variance was large (47.9%) and reflected the combined influence of all variables in the model. In the anxiety model, clinical features (55.9%) had two to three times greater explanatory power relative to personality (23.9%) and demographic features (15.2%), leaving only 5% of the remaining R² variance to combined effects. The F tests for specific variables in both models suggested that the removal of clinical factors (particularly PHQ-9) significantly deteriorated the predictive power of regression models. AUC statistics for the depression (0.67, SE = 0.02) and anxiety (0.74, SE = 0.02) prognostic indices applied to predict RCSI in the estimation sample were both statistically significant (p < .001); ROC curves are shown in Appendix B.

Validation of Case Complexity Model

PIs using the shrunken coefficients derived from the estimation sample were applied in the validation sample, yielding stable and statistically significant (p < .001) AUC estimates for depression (0.64, SE = 0.02) and anxiety (0.70, SE = 0.02) measures (see ROC curves in Appendix B). Overall, 28.6% of all patients were classified as Cx by the prognostic classification rule derived using ROC curve analyses. The proportion of Cx cases was lower in the subsample of patients who only accessed LIT (15.9%) by comparison to those who accessed LIT + HIT (37.3%) and those who only accessed HIT (36.7%); χ²(2) = 97.05, p < .001.

As illustrated in Figure 1, logistic regression models (see Table 2) confirmed that St cases were significantly more likely to attain RCSI in depression (OR = 1.81) and anxiety (OR = 2.81) symptoms compared with Cx cases, after controlling for baseline severity.

Figure 1. Reliable and clinically significant improvement (RCSI) in cases classified as standard (St) and complex (Cx).

Validation of Prognostic Indices Applied Out-of-Sample-Using Logistic Regression

Case Complexity and Treatment Selection

Logistic regression models presented in Table 3 indicated that Cx cases had a significantly greater probability of RCSI in depression symptoms if they directly accessed HIT, by comparison to a standard stepped care pathway LIT + HIT (OR = 2.23, p = .01). There was also a trend indicating the same advantage of HIT for Cx cases in the anxiety model, although this did not reach statistical significance (OR = 1.74, p = .08). No significant differences were found between treatment pathways in the regression models applied to St cases. These analyses controlled for baseline symptom severity and propensity scores (derived from logistic regression model in Appendix C). The results for the depression outcomes are illustrated in Figure 2; where Cx cases that were initially assigned to HIT (optimal prognostic treatment assignment) had a 16.3% increased probability of RCSI by comparison to Cx who were assigned to a conventional stepped care pathway (LIT + HIT).

Logistic Regression Models Assessing Case Complexity and Treatment Selection

Figure 2. Reliable and clinically significant improvement (RCSI) in cases classified as standard (St) and complex (Cx) according to treatment pathway.

Clinical Judgment Versus Prognostic Models

The aggregated hit rate in the full sample indicated that clinicians’ treatment assignment decisions agreed with the prognostic strategy in 65.6% of cases. The TMaP score for the full sample, however, was low (κ = 0.09, SE = 0.02, p < .001). A closer examination of individual therapists’ treatment assignment decisions (N = 1,247 nested within 26 therapists) revealed considerable variability in their hit rates (range = 36.5% to 84.7%; M = 62.9, SD = 14.3) and TMaP scores (range = −0.27 to 0.44; M = 0.05, SD = 0.20). As shown in Figure 3, hit rates and TMaP scores were moderately correlated (r = .67, p < .001), and approximately 48% of therapists had TMaP scores <0.

Figure 3. Distribution of hit rates and treatment matching precision (TMaP) scores across 26 therapists.

Main Findings

This study set out to contribute to the understanding of case complexity in psychological care, in view of the limited conceptual clarity and evidence base surrounding this topic. Our findings demonstrate that (a) several patient characteristics have a cumulative effect on treatment outcomes, (b) it is possible to make reasonably accurate prognoses using this information, and (c) prognostic models can help us to operationalize case complexity in a way that is clinically useful. Cases classed as Cx (28.6%) on the basis of prognostic data tended to have significantly poorer outcomes after psychological treatment. Furthermore, Cx cases were two times (OR = 2.23) more likely to attain RCSI in depression symptoms if they were initially assigned to a high-intensity intervention instead of usual stepped care. A similar trend was observed for anxiety symptoms, although this did not reach statistical significance.

A Conceptual Bridge Between Prognosis and Case Complexity

These results lend support to the clinical notion that some cases are more difficult to treat due to various complicating factors (Ruscio & Holohan, 2006), although clinicians’ intuitions and treatment planning are often inconsistent with the evidence base (Garb, 2005). We found that treatment assignment decisions guided by clinical judgment were consistent with prognostic models in 65.6% of cases. This rate of agreement could be achieved by chance, or simply by mechanically following stepped care guidelines and assigning all cases initially to LIT, because the base rate of Cx cases is relatively low (under 30%). This was evidenced more clearly by examining the aggregated TMaP score (0.09) which was close to zero. Overall, the findings indicate that depression improvement (RCSI) rates for Cx cases could be significantly increased (by approximately 16.3%) if clinical judgment was supported by prognostic treatment selection models.

This gap between practice and science is perhaps accentuated by an unwieldy literature on the topic of prognosis in psychological care. Previous authors have attempted to synthesize findings across multiple studies to elucidate predictors of depression and anxiety outcomes (e.g., Driessen & Hollon, 2010; Haby, Donnelly, Corry, & Vos, 2006; Hamilton & Dobson, 2002; Keeley et al., 2008; Kessler et al., 2017; Licht-Strunk et al., 2007; Nilsen, Eisemann, & Kvernmo, 2013). Although some convergent findings are evident, meta-analytic reviews that privilege data from clinical trials are limited by typically small samples with sparse and heterogeneous prognostic variables, often gathered in highly selected participants (i.e., those with specific disorders) that may not be representative of complex cases seen in routine care (Chambless & Ollendick, 2001). Naturalistic cohort studies can offer informative evidence to complement findings from controlled trials, especially where multiple variables are measured systematically across large health care populations, as exemplified in this study. Several such studies are yielding replicated findings (e.g., Beard et al., 2016; Delgadillo, Moreea, & Lutz, 2016; Delgadillo, Dawson, Gilbody, & Böhnke, 2017; Firth, Barkham, Kellett, & Saxon, 2015; Goddard, Wingrove, & Moran, 2015; Licht-Strunk et al., 2009).

Overall, the emerging literature on outcome prediction points to factors clustered around clinical (i.e., baseline symptom severity, diagnosis, comorbidity, functioning and disability, physical illnesses), demographic (i.e., age, ethnicity, employment, socioeconomic deprivation, marital status), characterological (i.e., personality disorder diagnoses or traits, interpersonal problems and style, trait anxiety and neuroticism), and dispositional domains (i.e., readiness to change, expectancy). Informed by advances in the biomedical literature (Safford et al., 2007; Shippee et al., 2012), we propose that complex cases in psychological care are characterized by the presence of measurable factors that map onto multiple domains (clinical, demographic, characterological and dispositional), which are statistically associated with clinical prognosis and have a cumulative—detrimental—effect on treatment outcomes. The concept of case complexity is, therefore, dimensional (i.e., degrees of complexity on a continuum), and complex cases can be distinguished from others using empirically derived population norms and classification rules.

Case complexity may challenge psychological improvement through several mechanisms. One possibility is that an accumulation of disadvantages (e.g., poverty, interpersonal difficulties, functional impairment, outgroup derogation due to minority ethnic status) could disrupt the balance between life stressors and coping resources (Shippee et al., 2012). Complexity could also interfere with adequate engagement with therapy, for example, by undermining expectancy, which is a well-established predictor of treatment outcomes (Constantino et al., 2011). Baseline severity is an important contributor to complexity, so another possibility is that high baseline severity does not completely block improvement but may dampen the effect of treatment (i.e., cases with high severity can attain reliable improvement even if their symptoms do not reach subclinical levels). Furthermore, our findings suggest that specific features (i.e., demographic, clinical, characterological) influence specific clinical outcomes (remission of depression, anxiety) differentially. For example, demographic factors (e.g., young age, unemployment) had a considerably larger influence over depression outcomes relative to clinical and characterological factors. Future research could focus on exploring the relative contribution of different prognostic domains to multiple outcome domains (symptoms, quality of life, functioning) and the mechanisms through which these cumulative disadvantages may complicate or undermine treatment.

Limitations

Some limitations should be considered when interpreting the results of this study. As is common in naturalistic data sets, we encountered several cases with missing data (>30%). To deal with this, we applied multiple imputation and sensitivity analyses that yielded similar prognostic models, albeit with some shrinkage observed in the imputed dataset. On this basis, it was appropriate to perform further validation analyses using cases with complete data, to simulate how prognostic assessments would be applied in routine care, where data imputation of missing values is unfeasible.

Another limitation concerning the data used in this study was that we only had access to pre-post outcome measures for the entire treatment pathway, and it was not possible to disaggregate the outcomes for LIT and HIT for cases that accessed both steps. However, we were able to determine that Cx that only accessed HIT tended to have better outcomes compared to those who accessed LIT + HIT (a lengthier and costly treatment pathway). This suggests that there are no benefits of having LIT sessions preceding HIT, and hence the advantage of being initially assigned to HIT may not be solely due to having a lengthier treatment episode. Previous research using more granular outcomes data for each treatment step suggested that cases with poor prognostic features had a higher probability of dropout and lower probability of improvement at the LIT step by comparison to HIT (Delgadillo et al., 2016). These emerging findings suggest that assigning complex cases directly to HIT seems justified, although future randomized controlled trials of this strategy are necessary to determine if it is indeed more cost-effective.

Other limitations include the lack of formal diagnostic assessments and the analysis of a limited number of prognostic variables. It is known, for example, that specific diagnoses such as post-traumatic stress disorder, eating disorders and obsessive–compulsive disorder are associated with poorer outcomes in stepped care services (Delgadillo et al., 2017), and it is plausible that such diagnoses could interact with other prognostic features. Notwithstanding these limitations, it is remarkable that this narrow range of variables yielded an accurate and clinically useful prognostic model. Other studies using routine practice data have shown that similar variables can be used to identify subgroups of cases with depression and anxiety problems that attain similar outcomes (Delgadillo et al., 2016; Lutz, Lowry, Kopta, Einstein, & Howard, 2001; Lutz et al., 2005; Saunders, Cape, Fearon, & Pilling, 2016).

Clinical Implications

In line with recent findings in stepped-care psychological treatment settings (Delgadillo et al., 2016; Lorenzo-Luaces, DeRubeis, van Straten, & Tiemens, 2017), the present study provides further evidence that applying prognostic indices to guide personalized treatment recommendations is likely to improve treatment outcomes. Low-intensity guided self-help interventions are recommended as first-line treatments for several common mental disorders (NICE, 2011) and are becoming widely available in routine stepped care services (Clark, 2011). The application of evidence-based treatment selection algorithms like the one demonstrated in this study could help to maximize the cost-effectiveness of LIT by selectively offering it to those who are most likely to derive benefits. Equally, prognostic models could be used to fast-track complex cases to HIT in a timely way.

Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., . . .Rush, J. D. (2006). The meta-analysis of clinical judgment project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34, 341–382. 10.1177/0011000005285875

Altman, D. G., & Bland, J. M. (1994). Diagnostic tests 3: Receiver operating characteristic plots. British Medical Journal, 309, 188. 10.1136/bmj.309.6948.188

Beard, C., Stein, A. T., Hearon, B. A., Lee, J., Hsu, K. J., & Björgvinsson, T. (2016). Predictors of depression treatment response in an intensive CBT partial hospital. Journal of Clinical Psychology, 72, 297–310. 10.1002/jclp.22269

Beck, J. S. (1998). Complex cognitive therapy treatment for personality disorder patients. Bulletin of the Menninger Clinic, 62, 170–194.

Bennett-Levy, J., Richards, D. A., & Farrand, P. (2010). Low intensity CBT interventions: A revolution in mental health care. In J.Bennett-Levy, D.Richards, P.Farrand, H.Christensen, K.Griffiths, D.Kavanagh, . . .C.Williams (Eds.), Oxford guide to low intensity CBT interventions (pp. 3–18). New York, NY: Oxford University Press. 10.1093/med:psych/9780199590117.003.0001

Boyd, C. M., Darer, J., Boult, C., Fried, L. P., Boult, L., & Wu, A. W. (2005). Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: Implications for pay for performance. Journal of the American Medical Association, 294, 716–724. 10.1001/jama.294.6.716

Chambless, D. L., & Ollendick, T. H. (2001). Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 52, 685–716. 10.1146/annurev.psych.52.1.685

Clark, D. M. (2011). Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: The IAPT experience. International Review of Psychiatry, 23, 318–327. 10.3109/09540261.2011.606803

Clark, D. M., Layard, R., Smithies, R., Richards, D. A., Suckling, R., & Wright, B. (2009). Improving access to psychological therapy: Initial evaluation of two UK demonstration sites. Behaviour Research and Therapy, 47, 910–920. 10.1016/j.brat.2009.07.010

Constantino, M. J., Arnkoff, D. B., Glass, C. R., Ametrano, R. M., & Smith, J. Z. (2011). Expectations. Journal of Clinical Psychology: In Session, 67, 184–192. 10.1002/jclp.20754

Delgadillo, J., Dawson, A., Gilbody, S., & Böhnke, J. R. (2017). Impact of long-term medical conditions on the outcomes of psychological therapy for depression and anxiety. The British Journal of Psychiatry, 210, 47–53. 10.1192/bjp.bp.116.189027

Delgadillo, J., Moreea, O., & Lutz, W. (2016). Different people respond differently to therapy: A demonstration using patient profiling and risk stratification. Behaviour Research and Therapy, 79, 15–22. 10.1016/j.brat.2016.02.003

Driessen, E., & Hollon, S. D. (2010). Cognitive behavioral therapy for mood disorders: Efficacy, moderators and mediators. The Psychiatric Clinics of North America, 33, 537–555. 10.1016/j.psc.2010.04.005

Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The 632+ bootstrap method. Journal of the American Statistical Association, 92, 548–560.

Firth, N., Barkham, M., Kellett, S., & Saxon, D. (2015). Therapist effects and moderators of effectiveness and efficiency in psychological wellbeing practitioners: A multilevel modelling analysis. Behaviour Research and Therapy, 69, 54–62. 10.1016/j.brat.2015.04.001

Garb, H. N. (2005). Clinical judgment and decision making. Annual Review of Clinical Psychology, 1, 67–89. 10.1146/annurev.clinpsy.1.102803.143810

Garfield, S. L. (1994). Research on client variables in psychotherapy. In A. E.Bergin & S. L.Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed.). New York, NY: Wiley.

Gifi, A. (1990). Nonlinear multivariate analysis. Chichester, England: Wiley.

Goddard, E., Wingrove, J., & Moran, P. (2015). The impact of comorbid personality difficulties on response to IAPT treatment for depression and anxiety. Behaviour Research and Therapy, 73, 1–7. 10.1016/j.brat.2015.07.006

Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical–statistical controversy. Psychology, Public Policy, and Law, 2, 293–323. 10.1037/1076-8971.2.2.293

Haby, M. M., Donnelly, M., Corry, J., & Vos, T. (2006). Cognitive behavioural therapy for depression, panic disorder and generalized anxiety disorder: A meta-regression of factors that may predict outcome. The Australian and New Zealand Journal of Psychiatry, 40, 9–19. 10.1080/j.1440-1614.2006.01736.x

Hamilton, K. E., & Dobson, K. S. (2002). Cognitive therapy of depression: Pretreatment patient predictors of outcome. Clinical Psychology Review, 22, 875–893. 10.1016/S0272-7358(02)00106-X

Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology/In Session, 61, 1–9. 10.1002/jclp.20108

Hawton, K., Salkovskis, P., Kirk, J., & Clark, D. M. (Eds.). (1989). Cognitive behaviour therapy for psychiatric problems: A practical guide. New York, NY: Oxford University Press. 10.1093/med:psych/9780192615879.001.0001

Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19. 10.1037/0022-006X.59.1.12

Keeley, M. L., Storch, E. A., Merlo, L. J., & Geffken, G. R. (2008). Clinical predictors of response to cognitive-behavioral therapy for obsessive-compulsive disorder. Clinical Psychology Review, 28, 118–130. 10.1016/j.cpr.2007.04.003

Kessler, R. C., van Loo, H. M., Wardenaar, K. J., Bossarte, R. M., Brenner, L. A., Ebert, D. D., . . .Zaslavsky, A. M. (2017). Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder. Epidemiology and Psychiatric Sciences, 26, 22–36. 10.1017/S2045796016000020

Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606–613. 10.1046/j.1525-1497.2001.016009606.x

Kroenke, K., Spitzer, R. L., Williams, J. B. W., Monahan, P. O., & Löwe, B. (2007). Anxiety disorders in primary care: Prevalence, impairment, comorbidity, and detection. Annals of Internal Medicine, 146, 317–325. 10.7326/0003-4819-146-5-200703060-00004

Licht-Strunk, E., van der Windt, D. A., van Marwijk, H. W. J., de Haan, M., & Beekman, A. T. F. (2007). The prognosis of depression in older patients in general practice and the community. A systematic review. Family Practice, 24, 168–180. 10.1093/fampra/cml071

Licht-Strunk, E., Van Marwijk, H. W. J., Hoekstra, T., Twisk, J. W. R., De Haan, M., & Beekman, A. T. F. (2009). Outcome of depression in later life in primary care: Longitudinal cohort study with three years’ follow-up. British Medical Journal, 338, a3079. 10.1136/bmj.a3079

Lorenzo-Luaces, L., DeRubeis, R. J., van Straten, A., & Tiemens, B. (2017). A prognostic index (PI) as a moderator of outcomes in the treatment of depression: A proof of concept combining multiple variables to inform risk-stratified stepped care models. Journal of Affective Disorders, 213, 78–85. 10.1016/j.jad.2017.02.010

Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., . . .Iveson, S. (2005). Predicting change for individual psychotherapy clients on the basis of their nearest neighbors. Journal of Consulting and Clinical Psychology, 73, 904–913. 10.1037/0022-006X.73.5.904

Lutz, W., Lowry, J., Kopta, S. M., Einstein, D. A., & Howard, K. I. (2001). Prediction of dose-response relations based on patient characteristics. Journal of Clinical Psychology, 57, 889–900. 10.1002/jclp.1057

Moran, P., Leese, M., Lee, T., Walters, P., Thornicroft, G., & Mann, A. (2003). Standardised Assessment of Personality–Abbreviated Scale (SAPAS): Preliminary validation of a brief screen for personality disorder. The British Journal of Psychiatry, 183, 228–232. 10.1192/bjp.183.3.228

Mundt, J. C., Marks, I. M., Shear, M. K., & Greist, J. H. (2002). The Work and Social Adjustment Scale: A simple measure of impairment in functioning. The British Journal of Psychiatry, 180, 461–464. 10.1192/bjp.180.5.461

Myhr, G., Talbot, J., Annable, L., & Pinard, G. (2007). Suitability for short-term cognitive behavioral therapy. Journal of Cognitive Psychotherapy, 21, 334–345. 10.1891/088983907782638743

National Institute for Health and Care Excellence (NICE). (2010). Depression: The treatment and management of depression in adults. London, UK: The British Psychological Society and The Royal College of Psychiatrists.

National Institute for Health and Care Excellence (NICE). (2011). Common mental health disorders: Identification and pathways to care. London, UK: Author. Retrieved from http://www.nice.org.uk.offcampus.lib.washington.edu/guidance/CG123

Nilsen, T. S., Eisemann, M., & Kvernmo, S. (2013). Predictors and moderators of outcome in child and adolescent anxiety and depression: A systematic review of psychological treatment studies. European Child & Adolescent Psychiatry, 22, 69–87. 10.1007/s00787-012-0316-3

Richards, D. A., & Borglin, G. (2011). Implementation of psychological therapies for anxiety and depression in routine practice: Two-year prospective cohort study. Journal of Affective Disorders, 133, 51–60. 10.1016/j.jad.2011.03.024

Roth, A. D., & Pilling, S. (2008). Using an evidence-based methodology to identify the competencies required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders. Behavioural and Cognitive Psychotherapy, 36, 129–147. 10.1017/S1352465808004141

Ruscio, A. M., & Holohan, D. R. (2006). Applying empirically supported treatments to complex cases: Ethical, empirical, and practical considerations. Clinical Psychology: Science and Practice, 13, 146–162. 10.1111/j.1468-2850.2006.00017.x

Safford, M. M., Allison, J. J., & Kiefe, C. I. (2007). Patient complexity: More than comorbidity. the vector model of complexity. Journal of General Internal Medicine, 22(Suppl. 3), 382–390. 10.1007/s11606-007-0307-0

Saunders, R., Cape, J., Fearon, P., & Pilling, S. (2016). Predicting treatment outcome in psychological treatment services by identifying latent profiles of patients. Journal of Affective Disorders, 197, 107–115. 10.1016/j.jad.2016.03.011

Schafer, J. L., & Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst’s perspective. Multivariate Behavioral Research, 33, 545–571. 10.1207/s15327906mbr3304_5

Schaink, A. K., Kuluski, K., Lyons, R. F., Fortin, M., Jadad, A. R., Upshur, R., & Wodchis, W. P. (2012). A scoping review and thematic classification of patient complexity: Offering a unifying framework. Journal of Comorbidity, 2, 1–9. 10.15256/joc.2012.2.15

Shippee, N. D., Shah, N. D., May, C. R., Mair, F. S., & Montori, V. M. (2012). Cumulative complexity: A functional, patient-centered model of patient complexity can improve research and practice. Journal of Clinical Epidemiology, 65, 1041–1051. 10.1016/j.jclinepi.2012.05.005

Smith, S. M., & O’Dowd, T. (2007). Chronic diseases: What happens when they come in multiples?The British Journal of General Practice, 57, 268–270.

Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166, 1092–1097. 10.1001/archinte.166.10.1092

Tarrier, N. (Ed.). (2006). Case formulation in cognitive behaviour therapy: The treatment of challenging and complex clinical cases. London, UK: Routledge.

Tarrier, N., Wells, A., & Haddock, G. (Eds.). (1998). Treating complex cases: The cognitive behavioural therapy approach. Chichester, UK: Wiley.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Methodological, 58, 267–288.

Tinetti, M. E., Bogardus, S. T., Jr., & Agostini, J. V. (2004). Potential pitfalls of disease-specific guidelines for patients with multiple conditions. The New England Journal of Medicine, 351, 2870–2874. 10.1056/NEJMsb042458

van Straten, A., Tiemens, B., Hakkaart, L., Nolen, W. A., & Donker, M. C. (2006). Stepped care vs. matched care for mood and anxiety disorders: A randomized trial in routine practice. Acta Psychiatrica Scandinavica, 113, 468–476. 10.1111/j.1600-0447.2005.00731.x

APPENDIX A: Penalized Categorical Regression Models With Optimal Scaling

ANOVA for PHQ-9 Model (Estimation Sample)

Figure A1. Lasso paths for Patient Health Questionnaire–9 (PHQ-9) model. See the online article for the color version of this figure.

Lasso-Based Model Coefficients

Figure A2. Categorical quantifications and transformation plots for Patient Health Questionnaire–9 (PHQ-9) model (significant predictors only).

ANOVA for GAD-7 Model (Estimation Sample)

Figure A3. Lasso paths for Generalized Anxiety Disorder–7 Scale (GAD-7) model. See the online article for the color version of this figure.

Lasso-Based Model Coefficients

Figure A4. Categorical quantifications and transformation plots for Generalized Anxiety Disorder–7 Scale (GAD-7) model (significant predictors only).

ANOVA for PHQ-9 Model (Estimation Sample)

Figure A1. Lasso paths for Patient Health Questionnaire–9 (PHQ-9) model. See the online article for the color version of this figure.

Lasso-Based Model Coefficients

Figure A2. Categorical quantifications and transformation plots for Patient Health Questionnaire–9 (PHQ-9) model (significant predictors only).

ANOVA for GAD-7 Model (Estimation Sample)

Figure A3. Lasso paths for Generalized Anxiety Disorder–7 Scale (GAD-7) model. See the online article for the color version of this figure.

Lasso-Based Model Coefficients

Figure A4. Categorical quantifications and transformation plots for Generalized Anxiety Disorder–7 Scale (GAD-7) model (significant predictors only).

APPENDIX B: Receiver Operating Characteristic (ROC) Curves

Figure B1. Patient Health Questionnaire–9 (PHQ-9) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Estimation sample.

Figure B2. Patient Health Questionnaire–9 (PHQ-9) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Validation sample.

Figure B3. Generalized Anxiety Disorder–7 scale (GAD-7) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Estimation sample.

Figure B4. Generalized Anxiety Disorder–7 scale (GAD-7) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Validation sample.

Figure B1. Patient Health Questionnaire–9 (PHQ-9) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Estimation sample.

Figure B2. Patient Health Questionnaire–9 (PHQ-9) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Validation sample.

Figure B3. Generalized Anxiety Disorder–7 scale (GAD-7) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Estimation sample.

Figure B4. Generalized Anxiety Disorder–7 scale (GAD-7) prognostic index (PI) as a predictor of post-treatment reliable and clinically significant improvement: Validation sample.

APPENDIX C: Logistic Regression Model Predicting (High-Intensity Treatment vs. Low-Intensity Treatment) Group Membership

Variables in the Equation

Variables in the Equation

Submitted: November 14, 2016 Revised: May 16, 2017 Accepted: May 29, 2017

Childhood Trauma and Personality Disorder Criterion Counts: A Co-twin Control Analysis

Acknowledgement: Supported in part by NIH Grant MH-068643 and grants from the Norwegian Research Council, the Norwegian Foundation for Health and Rehabilitation, the Norwegian Council for Mental Health, and the European Commission under the program “Quality of Life and Management of the Living Resources” of the Fifth Framework Program (QLG2-CT-2002-01254).

Stressful and potentially traumatic life events have been implicated in theories of psychopathology etiology and maintenance (Cicchetti & Cohen, 1997; Linehan, 1993). Exposure to potentially traumatic events during childhood has been related to a number of personality disorder (PD) diagnoses and symptoms (Battle et al., 2004), with Borderline PD receiving the most empirical attention. For example, individuals with clinical or subclinical Borderline PD symptoms are more likely to endorse having experienced childhood abuse compared to nonclinical controls (Bandelow et al., 2005; Laporte, Paris, Guttman, & Russell, 2011; Sansone, Hahn, Dittoe, & Wiederman, 2011). Additionally, etiologic theories of Borderline PD highlight the importance of the role of childhood trauma (CT; Linehan, 1993).

Associations between CT and other PDs also have been documented. For example, childhood maltreatment and abuse are related to Schizotypal PD symptoms (Berenbaum, Thompson, Milanek, Boden, & Bredemeier, 2008; Powers, Thomas, Ressler, & Bradley, 2011), specifically paranoia and unusual perceptual experiences (Steel, Marzillier, Fearon, & Ruddle, 2009). Childhood abuse and witnessing domestic violence have been associated with greater self-reported antisocial behaviors in adolescence (Sousa et al., 2011) and Antisocial PD symptoms in adulthood (Bierer et al., 2003). Other studies have reported broad associations between childhood abuse and the majority of categories of PD symptoms (Battle et al., 2004; Tyrka, Wyche, Kelly, Price, & Carpenter, 2009), even when participants were selected on the basis of having no history of Axis I psychopathology (Grover et al., 2007). Furthermore, childhood abuse and maltreatment are prospectively related to a range of PD symptoms and diagnoses (Cohen, Crawford, Johnson, & Kasen, 2005; Johnson, Cohen, Brown, Smailes, & Bernstein, 1999).

It is tempting to assume that the observed associations between CT and PDs are causal. Although associations have been well documented, a direct effect of CT on PDs has not been established. An alternative explanation is that common mechanisms explain significant covariation between CT and features of PDs (Chapman, Leung, & Lynch, 2008; New et al., 2009). The observed relationship between CT and PDs could also result from common environmental factors (e.g., stressful family environments) and/or shared genetic factors that predispose to both (Button, Scourfield, Martin, Purcell, & McGuffin, 2005; McGuigan & Pratt, 2001; Sartor et al., 2012). Indeed, likelihood of exposure to traumatic events is moderately influenced by genetic factors (Kendler & Baker, 2007), and it is possible that some of these same factors play a role in the development of PDs. Unique environmental influences may also play a role in the CT−PD relationship (e.g., impact of parental reactions on the trauma-exposed child; Nugent, Ostrowski, Christopher, & Delahanty, 2007). Individuals with PDs compared to those without PDs may also be more likely to report a history of CT because of greater negative emotionality, which may bias retrospective reporting (Hardt & Rutter, 2004).

Genetically informative samples consisting of twins who are discordant for CT may provide some insight into the role of CT in PDs, given that shared genetic and environmental factors may be accounted for statistically (Kendler & Campbell, 2009). For example, as traumatic events have been shown to correlate with multiple family background risk factors that are shared by twins (e.g., interpersonal loss, family discord, economic adversity), these factors are statistically controlled for in this model. Without sufficient control in epidemiological samples, which necessitate the measurement of all confounding factors, some of which are unknown, the clustering of traumatic events would likely serve to overestimate the association between CT and PDs. In the one study to our knowledge that addresses trauma and PDs using this design, Bornovalova and colleagues (2013) found that the relationship between childhood abuse and adult Borderline PD traits was likely noncausal and better accounted for by genetic factors. However, this study utilized a questionnaire to assess Borderline traits and did not assess a full range of CT events or PD categories. In fact, the heterogeneity of the assessment and definition of CT in the literature more broadly is problematic. Most notably, many studies fail to assess DSM–IV PTSD Criterion A for trauma exposure (Battle et al., 2004), resulting in a variable that could be assessing stressful life events or negative aspects of the family environment more generally. Past research also focuses largely on child maltreatment and abuse, without assessment or inclusion of other forms of CT that fall within the scope of DSM–IV Criterion A events (Grover et al., 2007). The current investigation sought to address several outstanding limitations of the existing literature by utilizing a genetically informed sample, employing a conservative definition of CT (i.e., DSM–IV PTSD Criterion A), allowing for inclusion of a broad range of CT event types, and using a clinical interview to assess CT and PD criterion counts.

The first aim of the current study was to detect and quantify an association between CT and PD criterion counts in a large sample of adult Norwegian twins obtained from the Norwegian Twin Registry (NTR). It was hypothesized that CT would evidence significant associations with PD criterion counts, above and beyond the effects of age, education level, and participant sex. Given that past studies consistently evidence sex differences in PD prevalence and expression (Torgersen, Kringlen, & Cramer, 2001; Verona, Sprague, & Javdani, 2012), we also evaluated an interaction between CT and participant sex in relation to PD criterion counts. The second aim of the study was to clarify the nature of an association between CT and PD criterion counts in a sample of twins discordant for CT. Specifically, we aimed to evaluate whether CT exerted a direct (i.e., potentially causal) or indirect (i.e., better accounted for by shared environmental and/or genetic factors) effect on PD criterion counts.

Sample and Assessment Method

The Norwegian National Medical Birth Registry, established on January 1, 1967, receives mandatory notification of all live births. The NTR identified and recruited twins from the registry, with participants completing questionnaire studies in 1992 (including twins born between 1967 and 1974) and 1998 (including twins born between 1967 and 1979). Of the 6,442 eligible twins that agreed to be contacted again after the second questionnaire, approximately 44% (2,794 twins) participated in an interview study initiated in 1999 (Tambs et al., 2009). This sample also included 68 twin pairs who had not completed the second questionnaire study, but were still recruited (due to technical problems). Data for the current investigation included all participants who completed the interview study and had complete data on PD criterion counts and CT (N = 2,780). This included a subset (n = 616) of twin pairs (46% monozygotic [MZ]) that were discordant for CT. Participants in the general sample (63.5% women) had a mean age of 28.2 (SD = 3.9) at the time of the interview and reported approximately 14.9 years of education (SD = 2.6).

Approval was received from The Norwegian Data Inspectorate and the Regional Ethical Committee approved the study. All participants provided written informed consent. Interviewers were primarily senior clinical psychology graduate students at the end of their 6-year training course (including at least 6 months of clinical practice; 75%) with the remainder (25%) being experienced psychiatric nurses, with the exception of two medical students. The interview training, conducted by one psychiatrist and two psychologists, consisted of a formal presentation on personality disorders, in-class demonstrations of the interview, multiple supervised role plays and test interviews, and group discussion of possible problems and scoring issues. The interviews, mostly face-to-face, were carried out between June, 1999 and May, 2004. For practical reasons, 231 interviews (8.3%) were done by phone. A different interviewer interviewed each twin in a pair.

Assessment of PDs

A Norwegian version of the Structured Interview for DSM–IV Personality (SIDP-IV; Pfohl, Blum, & Zimmerman, 1995), a comprehensive semistructured diagnostic interview, was used to assess all 10 DSM–IV PDs. The SIDP-IV has been successfully used in previous large-scale studies in Norway (Helgeland, Kjelsberg, & Torgersen, 2005; Torgersen et al., 2001). The SIDP-IV contains nonpejorative questions organized into topical sections rather than by individual PD to improve interview flow, and uses the “5-year rule,” meaning that behaviors, cognitions, and feelings that predominated for most of the past 5 years are judged to be representative of an individual’s personality. Each DSM–IV criterion is scored on a 4-point scale (0 = absent, 1 = subthreshold, 2 = present, or 3 = strongly present). Only the A criterion was assessed for Antisocial PD, given the 5-year assessment timeframe (i.e., Criterion C—presence of conduct disorder prior to age 15—was not assessed).

Given the low base rate of PDs, ordinal symptom counts were created that reflect the number of positively endorsed criteria for each disorder. Results from multiple threshold tests of these 10 PDs indicate that the four response options scored as successive integers represent increasing levels of “severity” on a single continuum of liability (Reichborn-Kjennerud et al., 2007; Torgersen et al., 2008). Because few individuals endorsed (scored 2 or greater) most of the criteria for individual PDs, high criterion counts were infrequent. These low frequency, high symptom counts were collapsed so that variation for all PDs was represented as six ordinal categories. This approach has been successfully utilized in past research using the current sample (Reichborn-Kjennerud et al., 2007). Previous studies using the current data have reported high interrater reliability (range of intraclass correlations for endorsed criterion counts = .81−.96) for the assessed PDs obtained by two raters (one psychologist and one psychiatrist interview trainer) scoring 70 audiotaped interviews (Kendler et al., 2008).

Assessment of CT

A Norwegian computerized version of the Munich-Composite International Diagnostic Interview (M-CIDI; (Wittchen & Pfister, 1997), a comprehensive structured diagnostic interview assessing Axis I diagnoses, was administered. The M-CIDI has good test−retest and interrater reliability (Wittchen, 1994; Wittchen, Lachner, Wunderlich, & Pfister, 1998). In the PTSD module of the M-CIDI, participants were asked if they had personally experienced or witnessed any of the following traumas: 1) a terrible experience at war, 2) serious physical threat (with a weapon), 3) rape, 4) sexual abuse as a child, 5) a natural catastrophe, 6) a serious accident, 7) being imprisoned, taken hostage, or kidnapped, or 8) another event. We defined CT as an event occurring before the age of 17 that met DSM–IV PTSD Criteria A1 (i.e., “the person experienced, witnessed, or was confronted with an event or events that involved actual or threatened death or serious injury, or a threat to the physical integrity of self or others) and A2 (i.e., “the person’s response involved intense fear, helplessness, or horror”; American Psychiatric Association, 1994). Approximately 17% (n = 467) of the total sample met these criteria.

Individuals’ worst CT were: 35.0% childhood sexual assault, 15.8% rape, 13.1% an accident, 12.6% an “other” traumatic event, 10.9% physical threat to oneself, 10.9% witnessing a traumatic event, 1.1% a natural disaster, and 0.5% being held hostage.

Data Analytic Plan

Analyses for the current study were conducted in SAS. First, the association between CT and PD criterion counts was examined in the general sample. A series of linear regression models was conducted, with age, education level, participant sex (1 = male, 2 = female), and CT (1 = no CT, 2 = CT) entered in level 1. To examine potential sex differences in the relationship between CT and PD criterion counts, the interaction of participant sex and CT was entered at level 2. A square root transformation was conducted for all PD criterion counts prior to inclusion in the regression models.

Second, a series of fixed effects regressions was conducted to examine CT−PD criterion count relationships among the subset of twin pairs discordant for CT, with twin pair serving as the fixed between-groups factor. This method allows for statistical control of unobserved between-family variation (e.g., genetic factors, family stressors, parenting style, socioeconomic factors, etc.). The relationships between CT and PD criterion counts were compared in the general and discordant twin samples. Based on such comparison, one may determine whether the observed relationship between CT and PD criterion counts in the general sample is likely direct or indirect (i.e., better accounted for by familial factors). Specifically, if the magnitude of the relationship between CT and PD criterion counts were comparable in the general and discordant twin samples, the effect is likely to be direct. If the relationship is significantly lesser in magnitude or nonexistent in the discordant twin sample, the effect is likely to be indirect, or accounted for by shared familial factors. The current investigation was not sufficiently powered to examine discordant MZ and DZ twin pairs separately, which would allow for speculation regarding whether shared environmental or genetic factors were responsible for observed indirect effects. For a more comprehensive overview of the cotwin control design, see K.S. Kendler et al.(1993). A total of 20 regression models (10 in the full sample, 10 in the discordant twin subsample) were conducted. Bonferroni correction for multiple testing indicated statistical significance at p = .003.

Descriptive Statistics and Zero-Order Correlations (General Sample)

See Table 1. Sex was significantly related to CT, with men being more likely to endorse a CT. Men also were more likely to endorse a greater number of criteria for Narcissistic, Antisocial, and Obsessive-Compulsive PDs, while women were more likely to endorse criteria for Schizotypal, Histrionic, Borderline, and Dependent PDs. Age and years of education were significantly related to various PD criterion counts with no particular pattern being observed. CT was significantly, yet modestly, related to a greater number of criteria for all PDs, with the exception of Avoidant PD.

Descriptive Statistics and Zero-Order Correlations in the Total Sample

Trauma Exposure and PD Criterion Counts

See Table 2 for regression statistics for the main effect of CT on PD criterion counts in the full and discordant twin samples. CT was significantly related to all PD criterion counts in the full sample, after covarying for sex, age, and education level, with the exception of Schizoid, Avoidant, and Dependent PDs. The greatest effects were observed for Borderline (r² = .013), Obsessive-Compulsive (r² = .008), Schizotypal (r² = .007), and Antisocial PDs (r² = .007; see Figure 1 for a comparison of effect sizes for all PD criterion counts). An interaction between CT and sex was statistically significant for Schizoid (β = .25, t = 2.95, p = .003) and Schizotypal PDs (β = .26, t = 3.16, p = .002). CT was significantly, yet modestly, related to Schizoid and Schizotypal PD criterion counts among women (β = .10, t = 4.10, sr² = .01, p < .001; and β = .13, t = 5.31, sr² = .02, p < .001, respectively) but not men (β = −.01, t = −.44, p = .661; and β = .01, t = .42, p = .678, respectively).

Childhood Trauma Predicting Personality Disorder Criterion Counts

Figure 1. Magnitude of relationship between childhood trauma and personality disorder criterion counts. Note: * p < .003, indicating statistical significance after Bonferroni correction for multiple testing within the indicated sample; analyses in the full sample represent the effect of childhood trauma on criterion counts above and beyond the variance accounted for by participant age, education level, and sex.

Results of the fixed effects regressions indicated that after accounting for family and genetic factors in the discordant twin sample, the relationship between CT and PD criterion counts was quite modest (see Table 2). Overall, the magnitude of the effect of CT on PD criterion counts was quite small prior to accounting for familial liability, not accounting for more than 1% of variance for any given disorder, and upon controlling for familial factors, the relationship between CT and PD criterion counts was essentially nonexistent (see Figure 1 for a comparison of effect sizes in the two samples). The potential exceptions to this pattern are for Borderline and Antisocial PD criterion counts, for which CT appears to account for a slight proportion of unique variation (<1%) beyond shared familial liability.

Our first aim was to examine associations between CT and PD criterion counts in the general sample. Consistent with past work, CT was significantly associated with the majority of PD criterion counts, after accounting for sex, age, and education (Battle et al., 2004; Grover et al., 2007; Tyrka et al., 2009). Significant associations were not detected for Schizoid, Avoidant, or Dependent PDs. The strongest associations were detected for Borderline, Obsessive-Compulsive, Schizotypal, and Antisocial PD criterion counts; however, CT accounted for less than 2% of variance in these disorders, even without the inclusion of rigorous covariates (e.g., life stressors, parenting style, etc.). In contrast, trait-level neuroticism accounts for approximately 45% of variance in Borderline PD traits prior to accounting for shared genetic factors (Distel et al., 2009).

The relationship between CT and Schizoid and Schizotypal PD criterion counts varied by sex, with a significant trauma−PD association being detected among women but not men. It is possible that CT is associated with factors promoting a decreased capacity for or interest in close relationships among women. However, the magnitude of effects was consistent with the pattern observed in the full sample, with CT accounting for no more than 2% of variance in these disorders. Prior to controlling for the role of familial factors, CT appears to exert quite modest influence on PD criterion counts.

Second, we sought to explain the nature of the observed effects through the use of a cotwin control design, which relies on comparing the magnitude of effects in the general sample with that observed in twins discordant for CT. The relationships between CT and Antisocial and Narcissistic PDs were comparable in the two samples. It is possible that the very small (i.e., r² < .01) effect of CT is causal in nature for these disorders. CT also was significantly related to Borderline PD criterion counts in the discordant twin sample; although, the magnitude of the effect in the discordant twins was 50% of that observed in the full sample, suggesting that a substantial portion of the relationship is likely better accounted for by familial factors rather than a direct effect of CT. The role of CT in the remaining PD criterion counts was essentially nonexistent in the discordant twin sample. It does not appear that CT plays a substantial role in the development of PD symptoms.

These findings are at odds with existing theories and clinical intuition regarding PD etiology, particularly in the case of Borderline PD. For example, one review paper on CT in Borderline PD concluded that, “the evidence suggests that childhood trauma should be included in a multifactorial model of BPD” (Ball & Links, 2009). It is worth noting that the studies reviewed by the authors were purely correlational and did not account for the role of familial factors. Marsha Linehan cites CT as a “prototypic invalidating experience” contributing to a biosocial model of Borderline PD etiology, with an entire stage of treatment being devoted to addressing CT and reducing trauma-related behaviors in Dialectical Behavior Therapy (Linehan, 1993). Finally, there has been empirical interest in discovering a biological link between CT and PDs. It has been suggested that CT may lead to neurobiological alterations (e.g., disruption of serotonin function), which in turn may lead to emotional and behavioral deficits in PDs (Lee, 2006). The current findings would suggest that focus on CT as a key risk factor in PDs may not be particularly fruitful. Indeed, this study replicates and extends the only other investigation to our knowledge that addresses CT and PDs using a discordant twin design, in which little to no direct relationship between CT and Borderline PD symptoms was detected (Bornovalova et al., 2013). Investigation of traumatic events in the etiology of Axis I psychopathology using genetically informed designs is needed.

The current project has a number of limitations. First, the base rate of high criterion counts in the current sample was quite low, preventing analysis at the diagnostic level. Future studies utilizing large samples and diverse methodologies are needed. Second, this study was underpowered to examine trauma type or severity in relation to PD criterion counts. Similarly, this study did not have data on familial response to CT. Parental response to trauma may influence the trauma-exposed child’s unique environment. For example, parental posttraumatic stress disorder symptoms and general parental distress may exert unique influence on children’s reactions to trauma (Nugent et al., 2007). Future studies investigating trauma characteristics and responses of the individual and his or her family to CT may be useful. Third, the ethnic and age composition of the current sample is relatively homogenous. Fourth, this study relied on retrospective reporting of CT. However, one would expect this reporting method to inflate the relationship between CT and PD symptoms; therefore, it is possible that the very modest effects detected in the current sample are upwardly biased by recall effects. Fifth, this study did not include a social desirability measure, which may be useful for future investigations in order to account for potential reporting bias. Sixth, this study was underpowered to analyze MZ and DZ twins separately, which would have provided additional information on whether genetic or shared environmental factors accounted for the relationship between CT and PD criterion counts. Finally, this study did not assess for a history of conduct disorder symptoms as related to Antisocial PD. Despite these limitations, the current study provides novel data on the relationship between CT and PD criterion counts.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Ball, J. S., & Links, P. S. (2009). Borderline personality disorder and childhood trauma: Evidence for a causal relationship. Current Psychiatry Reports, 11, 63–68. doi:10.1007/s11920-009-0010-4

Bandelow, B., Krause, J., Wedekind, D., Broocks, A., Hajak, G., & Ruther, E. (2005). Early traumatic life events, parental attitudes, family history, and birth risk factors in patients with borderline personality disorder and healthy controls. Psychiatry Research, 134, 169–179. doi:10.1016/j.psychres.2003.07.008

Battle, C. L., Shea, M. T., Johnson, D. M., Yen, S., Zlotnick, C., Zanarini, M. C.,. . .Morey, L. C. (2004). Childhood maltreatment associated with adult personality disorders: Findings from the Collaborative Longitudinal Personality Disorders Study. Journal of Personality Disorders, 18, 193–211. doi:10.1521/pedi.18.2.193.32777

Berenbaum, H., Thompson, R. J., Milanek, M. E., Boden, M. T., & Bredemeier, K. (2008). Psychological trauma and schizotypal personality disorder. Journal of Abnormal Psychology, 117, 502–519. doi:10.1037/0021-843X.117.3.502

Bierer, L. M., Yehuda, R., Schmeidler, J., Mitropoulou, V., New, A. S., Silverman, J. M., & Siever, L. J. (2003). Abuse and neglect in childhood: Relationship to personality disorder diagnoses. CNS Spectrums, 8, 737–754.

Bornovalova, M. A., Huibregtse, B. M., Hicks, B. M., Keyes, M., McGue, M., & Iacono, W. (2013). Tests of a direct effect of childhood abuse on adult borderline personality traits: A longitudinal discordant twin design. Journal of Abnormal Psychology, 122, 180–194. doi:10.1037/a0028328

Button, T. M., Scourfield, J., Martin, N., Purcell, S., & McGuffin, P. (2005). Family dysfunction interacts with genes in the causation of antisocial symptoms. Behavior Genetics, 35, 115–120. doi:10.1007/s10519-004-0826-y

Chapman, A. L., Leung, D. W., & Lynch, T. R. (2008). Impulsivity and emotion dysregulation in borderline personality disorder. Journal of Personality Disorders, 22, 148–164. doi:10.1521/pedi.2008.22.2.148

Cicchetti, D., & Cohen, D. (1997). Perspectives on developmental psychopathology. In S. S.Luthar, J. A.Burak, D.Cicchetti & J. R.Weisz (Eds.), Developmental psychopathology: Perspectives on adjustment, risk, and disorder (pp. 3–20). Cambridge, UK: Cambridge University Press.

Cohen, P., Crawford, T. N., Johnson, J. G., & Kasen, S. (2005). The Children in the Community Study of developmental course of personality disorder. Journal of Personality Disorders, 19, 466–486. doi:10.1521/pedi.2005.19.5.466

Distel, M. A., Trull, T. J., Willemsen, G., Vink, J. M., Derom, C. A., Lynskey, M., . . .Boomsma, D. I. (2009). The five-factor model of personality and borderline personality disorder: A genetic analysis of comorbidity. Biological Psychiatry, 66, 1131–1138. doi:10.1016/j.biopsych.2009.07.017

Grover, K. E., Carpenter, L. L., Price, L. H., Gagne, G. G., Mello, A. F., Mello, M. F., & Tyrka, A. R. (2007). The relationship between childhood abuse and adult personality disorder symptoms. Journal of Personality Disorders, 21, 442–447. doi:10.1521/pedi.2007.21.4.442

Hardt, J., & Rutter, M. (2004). Validity of adult retrospective reports of adverse childhood experiences: Review of the evidence. Journal of Child Psychology and Psychiatry, 45, 260–273. doi:10.1111/j.1469-7610.2004.00218.x

Helgeland, M. I., Kjelsberg, E., & Torgersen, S. (2005). Continuities between emotional and disruptive behavior disorders in adolescence and personality disorders in adulthood. The American Journal of Psychiatry, 162, 1941–1947. doi:10.1176/appi.ajp.162.10.1941

Johnson, J. G., Cohen, P., Brown, J., Smailes, E. M., & Bernstein, D. P. (1999). Childhood maltreatment increases risk for personality disorders during early adulthood. Archives of General Psychiatry, 56, 600–606. doi:10.1001/archpsyc.56.7.600

Kendler, K. S., Aggen, S. H., Czajkowski, N., Roysamb, E., Tambs, K., Torgersen, S., et alReichborn-Kjennerud, T. (2008). The structure of genetic and environmental risk factors for DSM–IV personality disorders: A multivariate twin study. Archives of General Psychiatry, 65, 1438–1446. doi:10.1001/archpsyc.65.12.1438

Kendler, K. S., & Baker, J. H. (2007). Genetic influences on measures of the environment: A systematic review. Psychological Medicine, 37, 615–626. doi:10.1017/S0033291706009524

Kendler, K. S., & Campbell, J. (2009). Interventionist causal models in psychiatry: Repositioning the mind-body problem. Psychological Medicine, 39, 881–887. doi:10.1017/S0033291708004467

Kendler, K. S., Neale, M. C., MacLean, C. J., Heath, A. C., Eaves, L. J., & Kessler, R. C. (1993). Smoking and major depression: A causal analysis. Archives of General Psychiatry, 50, 36–43. doi:10.1001/archpsyc.1993.01820130038007

Laporte, L., Paris, J., Guttman, H., & Russell, J. (2011). Psychopathology, childhood trauma, and personality traits in patients with borderline personality disorder and their sisters. Journal of Personality Disorders, 25, 448–462. doi:10.1521/pedi.2011.25.4.448

Lee, R. (2006). Childhood trauma and personality disorder: Toward a biological model. Current Psychiatry Reports, 8, 43–52. doi:10.1007/s11920-006-0080-5

Linehan, M. M. (1993). Cognitive-behavioral treatment of borderline personality disorder. New York, NY: Guilford Press.

McGuigan, W. M., & Pratt, C. C. (2001). The predictive impact of domestic violence on three types of child maltreatment. Child Abuse & Neglect, 25, 869–883. doi:10.1016/S0145-2134(01)00244-7

New, A. S., Fan, J., Murrough, J. W., Liu, X., Liebman, R. E., Guise, K. G., et alCharney, D. S. (2009). A functional magnetic resonance imaging study of deliberate emotion regulation in resilience and posttraumatic stress disorder. Biological Psychiatry, 66, 656–664. doi:10.1016/j.biopsych.2009.05.020

Nugent, N. R., Ostrowski, S., Christopher, N. C., & Delahanty, D. L. (2007). Parental posttraumatic stress symptoms as a moderator of child’s acute biological response and subsequent posttraumatic stress symptoms in pediatric injury patients. Journal of Pediatric Psychology, 32, 309–318. doi:10.1093/jpepsy/jsl005

Pfohl, B., Blum, N., & Zimmerman, M. (1995). Structured Interview for DSM–IVPersonality: SIDP-IV. Iowa City, IA: University of Iowa, Department of Psychology.

Powers, A. D., Thomas, K. M., Ressler, K. J., & Bradley, B. (2011). The differential effects of child abuse and posttraumatic stress disorder on schizotypal personality disorder. Comprehensive Psychiatry, 52, 438–445. doi:10.1016/j.comppsych.2010.08.001

Reichborn-Kjennerud, T., Czajkowski, N., Neale, M. C., Orstavik, R. E., Torgersen, S., Tambs, K., et alKendler, K. S. (2007). Genetic and environmental influences on dimensional representations of DSM–IV cluster C personality disorders: A population-based multivariate twin study. Psychological Medicine, 37, 645–653. doi:10.1017/S0033291706009548

Sansone, R. A., Hahn, H. S., Dittoe, N., & Wiederman, M. W. (2011). The relationship between childhood trauma and borderline personality symptomatology in a consecutive sample of cardiac stress test patients. International Journal of Psychiatry in Clinical Practice, 15, 275–279. doi:10.3109/13651501.2011.593263

Sartor, C. E., Grant, J. D., Lynskey, M. T., McCutcheon, V. V., Waldron, M., Statham, D. J.,. . .Martin, N. G. (2012). Common heritable contributions to low-risk trauma, high-risk trauma, posttraumatic stress disorder, and major depression. Archives of General Psychiatry, 69, 293–299. doi:10.1001/archgenpsychiatry.2011.1385

Sousa, C., Herrenkohl, T. I., Moylan, C. A., Tajima, E. A., Klika, J. B., Herrenkohl, R. C., & Russo, M. J. (2011). Longitudinal study on the effects of child abuse and children’s exposure to domestic violence, parent-child attachments, and antisocial behavior in adolescence. Journal of Interpersonal Violence, 26, 111–136. doi:10.1177/0886260510362883

Steel, C., Marzillier, S., Fearon, P., & Ruddle, A. (2009). Childhood abuse and schizotypal personality. Social Psychiatry and Psychiatric Epidemiology, 44, 917–923. doi:10.1007/s00127-009-0038-0

Tambs, K., Ronning, T., Prescott, C. A., Kendler, K. S., Reichborn-Kjennerud, T., Torgersen, S., & Harris, J. R. (2009). The Norwegian Institute of Public Health twin study of mental health: Examining recruitment and attrition bias. Twin Research and Human Genetics, 12, 158–168. doi:10.1375/twin.12.2.158

Torgersen, S., Czajkowski, N., Jacobson, K., Reichborn-Kjennerud, T., Roysamb, E., Neale, M. C., & Kendler, K. S. (2008). Dimensional representations of DSM–IV cluster B personality disorders in a population-based sample of Norwegian twins: A multivariate study. Psychological Medicine, 38, 1617–1625. doi:10.1017/S0033291708002924

Torgersen, S., Kringlen, E., & Cramer, V. (2001). The prevalence of personality disorders in a community sample. Archives of General Psychiatry, 58, 590–596. doi:10.1001/archpsyc.58.6.590

Tyrka, A. R., Wyche, M. C., Kelly, M. M., Price, L. H., & Carpenter, L. L. (2009). Childhood maltreatment and adult personality disorder symptoms: Influence of maltreatment type. Psychiatry Research, 165, 281–287. doi:10.1016/j.psychres.2007.10.017

Verona, E., Sprague, J., & Javdani, S. (2012). Gender and factor-level interactions in psychopathy: Implications for self-directed violence risk and borderline personality disorder symptoms. Personality Disorders: Theory, Research, and Treatment, 3, 247–262. doi:10.1037/a0025945

Wittchen, H. U. (1994). Reliability and validity studies of the WHO–Composite International Diagnostic Interview (CIDI): A critical review. Journal of Psychiatric Research, 28, 57–84. doi:10.1016/0022-3956(94)90036-1

Wittchen, H. U., Lachner, G., Wunderlich, U., & Pfister, H. (1998). Test-retest reliability of the computerized DSM–IV version of the Munich-Composite International Diagnostic Interview (M-CIDI). Social Psychiatry and Psychiatric Epidemiology, 33, 568–578. doi:10.1007/s001270050095

Wittchen, H. U., & Pfister, H. (1997). DIA-X-Interviews (M-CIDI): Manual fur Screening-Verfahren und Interview; InterviewheftLangsschnittuntersuchung (DIA-X-Lifetime); Erganzungsheft (DIA-X lifetime); InterviewheftQuerschnittuntersuchung (DIA-X 12 Monate); Erganzungsheft (DIA-X 12 Monate); PC-ProgrammzurDurchfuhrung des Interviews (Langs-und Querschnittuntersuchung); Auswertungsprogramm. Frankfurt, Germany: Swets & Zeitlinger.

Submitted: September 16, 2012 Revised: July 23, 2013 Accepted: July 25, 2013

Chronic Sleep Disturbances and Borderline Personality Disorder Symptoms / BRIEF REPORT

Acknowledgement: Support for this project was provided, in part, by the Brain and Behavior Research Foundation and the Families for Borderline Personality Disorder Research with an early investigator grant (Edward A. Selby, principal investigator). The National Comorbidity Survey–Replication (NCS-R) was supported by National Institute of Mental Health Grant U01-MH60220, with supplemental support from the National Institute on Drug Abuse, the Substance Abuse and Mental Health Services Administration, the Robert Wood Foundation (Grant 044708), and the John W. Alden Trust. The views and opinions expressed in this report are those of the author and should not be construed to represent the views of any sponsoring organizations, agencies, or the U.S. government. A complete list of NCS publications and the full text of all NCS-R instruments can be found at http://www.hcp.med.harvard.edu/ncs. The NCS-R is carried out in conjunction with the World Health Organization World Mental Health (WMH) Survey Initiative.

Although not traditionally thought of as a disorder associated with sleep disturbances, there is growing evidence that those with borderline personality disorder (BPD) experience a variety of problems with sleep, including increased sleep onset latency and low sleep efficiency during polysomnography assessments (Bastien, Guimond, St-Jean, & Lemelin, 2008), abnormal sleep architecture (Battaglia, Strambi, Bertella, Bajo, & Bellodi, 1999), and nightmares (Asaad, Okasha, & Okasha, 2002; Selby, Ribeiro, & Joiner, in press). Sleep problems are clinically pertinent to BPD, as they are linked to functional impairment (Roth et al., 2006) and emotion dysregulation (Zohar, Tzischinsky, Epstein, & Lavie, 2005). To date, minimal research has systematically examined sleep disturbances in BPD, particularly chronic sleep problems. Chronic sleep problems involve difficulty sleeping most nights for an extended period of time (often lasting weeks to months), as opposed to acute sleep problems, which may last for a few days or arise intermittently, and can lead to major problems in daily functioning (Simon & Von Korff, 1997). Research is also lacking on BPD and the daytime consequences of chronic sleep problems, such as excessive daytime sleepiness, poor sleep-related fatigue, and difficulties engaging in activities due to poor sleep. Importantly, BPD may increase vulnerability to sleep problems, due to issues such as emotion dysregulation, and poor sleep may result in elevated daytime functional impairment.

Improving our understanding of sleep disturbances in BPD is also relevant to improving our interventions. At present, dialectical behavior therapy (DBT; Linehan, 1993) is the only psychotherapy for BPD that specifically addresses sleep problems. Improving the sleep of patients with BPD may aid in improving their ability to manage stressful situations, employ coping skills, and improve overall levels of energy and positive affect. In turn, improving the ability to manage stress may further reduce sleep problems (Carlson & Garland, 2005). Those with BPD also frequently experience suicidal ideation, the experience of which has itself also been linked to sleep problems (Sjöström, Wærn, & Hetta, 2007; Wojnar et al., 2009). This makes it important to determine if the sleep problems of those with BPD also occur beyond the context of suicidal ideation.

Previous studies on sleep and BPD have involved small samples, often during acute polysomnography studies, and none to date have examined BPD and chronic sleep problems in a large epidemiological sample. One advantage to using such a sample is reduced treatment-seeking bias and increased understanding of how these issues affect people in the community at-large. Another advantage is the ability to control for Axis I disorders that are intertwined with acute and chronic sleep problems (e.g., depression, anxiety disorders)—an important issue given the role of comorbidity in BPD (Lenzenweger, Lane, Loranger, & Kessler, 2007). The purpose of the present study was to examine chronic sleep disturbances, poor sleep-related consequences, and social/emotional and cognitive impairment as a function of poor sleep in those exhibiting BPD symptoms with the National Comorbidity Survey–Replication (NCS-R; Kessler & Merikangas, 2004).

Sample and Diagnostic Assessment

The NCS-R was a nationally representative, institutional review board–approved survey of adults age 18 and older designed to involve multistage clustered area probability sampling and conducted between 2001 and 2003. Overall, the survey consisted of 9,282 respondents, with an overall response rate of 70.9%. All respondents in the NCS-R completed the Part I diagnostic interview, which consisted of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI 3.0; Kessler & Üstün, 2004) and included diagnostic information on anxiety disorders, mood disorders, and substance use disorders. Blinded clinical reappraisals of a probability subsample of the NCS-R indicated good concordant validity between the Axis I diagnosis (Kessler et al., 2005) according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM–IV;American Psychiatric Association, 1994) and the CIDI diagnosis. In addition to completing assessment the CIDI, a probability subsample of respondents (N = 5,692), with and without DSM–IV diagnoses, also received the NCS-R Part II interview, which also assessed personality disorders and sleep problems; only NCS-R Part II data were used in this study.

BPD Symptoms

Respondents completed eight questions taken from the International Personality Disorder Examination (IPDE) screening questionnaire (Loranger et al., 1994) designed to measure BPD symptoms. These items have been used extensively in examining personality disorders (e.g., Lenzenweger, 1999; Lenzenweger et al., 2007) and have been found to significantly predict personality disorder diagnoses when the IPDE was clinician-administered (Loranger, 1999; Loranger et al., 1994). Furthermore, the more IPDE items were endorsed, the higher was the probability that a personality disorder diagnosis was obtained using structured clinical interviews (Lenzenweger, Loranger, Korfine, & Neff, 1997). Items were rated with a dichotomous yes (1) or no (0) answer, and all were summed to create a continuous measure of BPD (α = .74). Importantly, clinical reappraisal interviews were previously conducted by phone on a probability subsample of 214 respondents from Part II of the NCS-R and screened positive for personality disorder symptoms, and clinical reassessments with the IPDE were highly correlated (r > .90) with these items (Lenzenweger et al., 2007). Of note, because the NCS-R included a separate suicidal behavior assessment, the IPDE BPD question on suicidal and self-injurious behavior was not included. As a result of this missing criterion, three indices of BPD were generated for use in this study: (1) a dichotomous measure for those endorsing 5+ diagnostic criteria for BPD (to obtain odds ratios), (2) a continuous measure of BPD symptoms, and (3) a dichotomous BPD diagnosis (5+ symptoms endorsed) including occurrence of a lifetime suicide attempt (from the suicide assessment below).

Regarding the distribution of BPD symptoms, Selby (2013) found that the base rates (BRs) of BPD symptoms in this sample, as well as the endorsement of those symptoms by those with BPD (diagnostic sensitivity; DS), were well represented: intense anger (BR = 23%, DS = 73%), affective instability (BR = 30%, DS = 90%), chronic emptiness (BR = 21%, DS = 83%), identity disturbance (BR = 19%, DS = 73%), stress-related paranoia (BR = 17%, DS = 69%), avoiding abandonment (BR = 12%, DS = 60%), impulsivity (BR = 40%, DS = 89%), and unstable relationships (BR = 15%, DS = 60%). Thus, those with BPD in this sample exhibited the full spectrum of BPD symptoms.

Chronic Sleep Problems

Three common sleep problems assessed during Part II of the NCS-R were used (1) delayed sleep onset latency (SOL; “Nearly every night it took you two hours or longer before you could fall asleep”), (2) amount of time spent awake after sleep onset (WASO; “You woke up nearly every night and took an hour or more to get back to sleep”), and (3) waking earlier in the morning than desired (EMA; “You woke up nearly every morning at least two hours earlier than you wanted to”). Each was prefaced as happening for “periods lasting two weeks or longer in the past 12 months.” All were consistent with DSM–IV definitions for the associated sleep problem and have been reported on previously (Roth et al., 2006).

Poor Sleep Consequences

In addition to chronic sleep problems, this study examined the consequences of poor sleep using three assessments: (1) problems “feeling sleepy during the day,” (2) feeling fatigued during the day due to “poor sleep,” and (3) frequency of being “too tired to complete daily activities” as a result of poor sleep. All questions were assessed in the same section as the above questions about chronic sleep problems. The first question, on the experience of daytime sleepiness most days for two or more weeks in the past 12 months, was coded as yes (1) or no (0). The second question, relating to daytime fatigue as a result of poor sleep, and the third question, regarding frequency of being too tired to carry out daily activities—both of which were asked in terms of the frequency of the problem during the worst month in the past year—were rated on 4-point Likert scales. These two questions were originally rated 1 (never), 2 (rarely), 3 (sometimes), and 4 (often). In order to simplify the presentation of results and present odds ratios, these items were recoded to reflect that 4 (often experience of this problem) was recoded as (1), and all other ratings were coded as (0).

Functional Impairment

Six outcomes were utilized from the World Health Organization Disability Assessment Schedule II (WHO-DAS II; Chwastiak & Von Korff, 2003), a 36-item measure for general disability impairment. Respondents reported the severity of each problem over the last 30 days regarding the following: (1) days having been totally unable to work or carry out daily activities (days out of role), (2) days able to carry out normal activities but with reduced workload or inhibited productivity (reduced work quality), (3) difficulty caring for oneself with activities such as hygiene (self-care), (4) difficulties with physical mobility, (5) cognitive impairment (e.g., difficulty remember things), and (6) difficulty with social and emotional role performance (e.g., controlling emotions when around other people). The complex method (Chwastiak & Von Korff, 2003), which involves item response theory, was used to score each scale, and this resulted in a continuous scale range from 0 (No disability) to 100 (Full disability).

Comorbid DSM–IV Disorders

All respondents were rated for Axis I diagnoses with the CIDI, which has good concordance with other structured clinical interviews (Kessler & Üstün, 2004; Kessler et al., 2005). The following disorders, which were present over the last 12 months, were included as covariates in all analyses: major depressive disorder (MDD), dysthymia, manic episode, alcohol and drug dependence, panic disorder (with and without agoraphobia), generalized anxiety disorder (GAD), and posttraumatic stress disorder (PTSD). Each of these disorders has sleep problems as a symptom or has been linked to sleep problems, and their inclusion allowed for benchmarks to compare the association between BPD and sleep problems to. To examine comorbidity, variables were generated to indicate presence (1) or absence (0) of any two or three Axis I disorders.

Suicidality

As a part of the CIDI, all respondents completed a suicide risk assessment that included lifetime history of suicidal behavior as well as suicidal behavior in the last 12 months. Those who had “seriously thought about committing suicide” were coded and used as a covariate in analyses, and the presence of a lifetime suicide attempt was also used to generate one of the BPD indices described above.

Sociodemographic Control Variables

The following variables from the NCS-R were included in all analyses due to their potential impact on sleep problems: age, sex, race–ethnicity, education level, marital status, occupational status, family income level, and number of preschool children living at home. The relationships between covariates and personality disorders and sleep problems has previous been reported on (Lenzenweger et al., 2007; Roth et al., 2006). Also included was the presence or absence of any one of the following chronic health conditions endorsed, due to potential sleep interference: arthritis, neck or back ache, headaches, chronic pain, chronic allergies, stroke, and heart disease.

Logistic regression analyses were used to examine sleep problems and consequences, with the coefficients and their standard errors exponentiated as odds ratios with 95% confidence intervals. All analyses included medical and sociodemographic control variables, comorbid Axis I disorders, and presence of suicidal ideation in the last year; adjusted odds ratios (AORs) were presented. Finally, the interactions between BPD symptoms and chronic sleep problems in predicting functional impairment were examined with linear regression. Because the NCS-R was a complex sample involving clustering, stratification, and weighting specific to Part II, in order to adjust for potential differences in probability of selection for the sample, data analysis had to account for these procedures. Accordingly, standard errors of the logistic regression and linear regression analyses were adjusted for these sampling and weighting procedures with the COMPLEX function of the MPlus statistical program (Muthén & Muthén, 2008–2010).

Prevalence of Sleep Problems and Borderline Personality Disorder

A total of 63% of those meeting diagnostic criteria for BPD (5+ symptoms endorsed) reported having at least one of the sleep problems assessed. The average duration for sleep problems for those with BPD was 19.9 weeks (SD = 21.6), which was significantly more than for those without BPD (M = 8.9, SD = 17.2), F(1, 6590) = 45.4, p < .01, d = 0.60. As seen in Table 1, those with BPD reported significant experiences with delayed SOL (AOR = 1.8, wald = 33.3, p < .01), WASO (AOR = 1.9, wald = 41.3, p < .01), and EMA (AOR = 2.3, wald = 64.8, p < .01). BPD diagnosis was also a significant predictor of having all three chronic sleep problems (27% of those with BPD; wald = 37.75, p < .01, AOR = 2.1). BPD associations with each of the chronic sleep problems were similar in magnitude to those of disorders traditionally linked to sleep problems or involving sleep problems in diagnostic criteria (e.g., GAD, MDD, PTSD).

Multivariate Logistic Regression Analyses Predicting Sleep Problems (N = 5,692)

Borderline Personality Disorder and Poor Sleep-Related Consequences

Approximately 66% of those with BPD reported having at least one consequence over the last 12 months. BPD diagnosis (see Table 2) demonstrated clear and consistent associations, beyond covariates, with sleepiness during the day (AOR = 2.0, wald = 51.2, p < .01), daytime fatigue due to poor sleep (AOR = 2.1, wald = 43.4, p < .01), and being too tired to complete daily activities (AOR = 1.9, wald = 12.8, p < .01). BPD diagnosis was also a significant predictor of having all three poor sleep consequences (6% of those with BPD; wald = 7.5, p < .01, AOR = 2.1). As with chronic sleep problems, BPD diagnosis had AORs similar in magnitude to those of many Axis I disorders traditionally associated with sleep problems.

Multivariate Logistic Regression Analyses Predicting Poor Sleep Consequences (N = 5,692)

Functional Impairment Associated With Poor Sleep and Borderline Personality Symptoms

BPD interacted with sleep problems (see Table 3) to indicate more problems with self-care (β = .08, t = 3.0, p < .01), cognitive impairment (β = .17, t = 7.1, p < .001), and social/emotional impairment (β = .19, t = 7.9, p < .001) after accounting for key covariates. Figure 1 indicates worse impairments for those with more BPD symptoms who exhibited more sleep problems for these areas of functioning. No significant interactions were found for days out of role, role impairment, or decreased physical mobility.

BPD and Chronic Sleep Problems Predicting Functional Impairment

Figure 1. Interaction between BPD symptoms and chronic sleep problems on functional impairment on the World Health Organization Disability Assessment Schedule II. Low and high levels refer to one standard deviation below and above the mean, respectively. BPD = borderline personality disorder (referring to BDP symptoms); SLP = sleep (referring to the number of sleep problems).

The current study found a clear and consistent report of chronic sleep disturbances in those with BPD, even after accounting for key covariates, with many experiencing delayed SOL, increased WASO, and increased EMA most days for at least 2 weeks over the last year. Interestingly, the magnitudes of the AORs for BPD on chronic sleep problems was similar to those for other Axis I disorders often associated with sleep disturbance and with sleep-related criteria (i.e., MDD, GAD). BPD was also significantly associated, due to problems sleeping, with increased daytime sleepiness, fatigue, and feeling too tired to complete daytime activities. Potential reasons for the association between BPD and sleep problems, beyond Axis I contributions, may be that BPD psychopathology increases vulnerability to sleep problems because of emotion dysregulation or rumination (Selby & Joiner, 2009) or because of interpersonal conflicts during the day. Such experiences during the day may translate into difficulty falling asleep or frequent awakenings due to preoccupation with the problems or increased arousal. They may also experience frequent nightmares on days with more emotion dysregulation (Selby et al., in press).

Findings also indicated that when those with BPD have sleep problems they might experience increased difficulties with emotion dysregulation, problems in social relationships or self-care, and memory problems. These interactions suggest that when both problems are present a positive feedback loop may arise where BPD symptoms may contribute to poor sleep and poor sleep aggravates symptoms of BPD. However, there may be differential effects of sleep problems on aggravating BPD symptoms, and some may be more affected than others. For example, despite the finding that many with BPD reported feeling too tired to complete daily activities due to poor sleep, BPD symptoms did not interact with sleep problems to predict number of days out of role or reduced role quality, nor did the interaction predict decreased physical mobility. However, those with elevated BPD symptoms and sleep problems reported more cognitive and social/emotional impairment. These findings indicated that when those with BPD are experiencing sleep problems, they may have increased problems with issues such as emotion dysregulation, social relationships, and remembering things (potentially impacting techniques and skills learned in therapy). Furthermore, a significant interaction was also found for those with BPD and sleep problems in predicting decreased ability for self-care, and reduced self-care also likely worsens problems with emotional, social, and cognitive regulation. Future research should compare those with BPD who do and do not have chronic sleep problems to determine if there are significant group differences in intensity or duration of specific BPD symptoms or if symptoms are exacerbated at a more general level.

Important strengths of the current study to consider include use of a large, nontreatment-seeking sample and controlling for medical, Axis I, and sociodemographic variables associated with sleep problems. One primary limitation involved cross-sectional assessment of sleep problems over the last year (e.g., potential recall bias), and there was some temporal inconsistency between the indices measured over the course of the last year (sleep problems) and the indices measured over the last 30 days (functional impairment). Longitudinal studies are needed to determine if BPD increases vulnerability to sleep problems or if sleep problems simply aggravate BPD symptoms. Further, all indices in this study were self-report, and further studies with sleep diary monitoring and/or polysomnography studies may be needed to replicate and extend these findings and ascertain more precise assessments of chronic sleep problems in BPD. Another limitation was that BPD was assessed with IPDE screening items, and findings should be replicated in samples where BPD is assessed with structured clinical interviews. Finally, other factors may be involved in the BPD association with poor sleep, such as being hypervigilant or worried about sleep problems, poor sleep environment, or sleep state misperceptions.

Clinically, monitoring of sleep problems and associated impairment may be important in therapy, and it may be beneficial to thoroughly cover facets of sleep hygiene as a routine part of therapy for those with BPD. Although some therapies for BPD integrate sleep hygiene to some extent (e.g., the emotion regulation module of DBT; Linehan, 1993), many therapies may overlook this issue. It may be the case that treating sleep will help reduce negative emotion in those with BPD, and such daytime improvements may further improve overall sleep quality. Finally, sleep hygiene alone may not be enough to treat some BPD patients with chronic sleep problems, and some patients may also benefit from additional cognitive behavior therapy for insomnia (Edinger & Means, 2005).

¹ Diagnostic efficiency analyses of the IPDE BPD screening items have indicated adequate sensitivity (all >.60), specificity (all >.66), negative predictive power (all >.95), and total predictive value (all >.68); however, all were somewhat low on positive predictive power (range .24–.53), indicating that endorsing only one symptom was not a strong indicator for the presence of a BPD diagnosis and supporting the notion that multiple items needed to be endorsed for a diagnosis (Selby, 2013).

² The main effects of BPD diagnosis on the six indices of impairment measured by the WHO-DAS II have previously been reported on, with significant effects found for BPD diagnosis on mobility, cognition, days out of role, diminished role quality, and problems with social and emotional functioning (Lenzenweger et al., 2007). However, the current study builds on these findings by examining the interaction between a continuous predictor of BPD symptoms and the number of chronic sleep problems assessed, and in this study the WHO-DAS II scales were left as continuous variables rather than dichotomized as in the previous study.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Asaad, T., Okasha, T., & Okasha, A. (2002). Sleep EEG findings in ICD-10 borderline personality disorder in Egypt. Journal of Affective Disorders, 71, 11–18. doi:10.1016/S0165-0327(01)00357-3

Bastien, C. H., Guimond, S., St-Jean, G., & Lemelin, S. (2008). Signs of insomnia in borderline personality disorder. Journal of Clinical Sleep Medicine, 4, 462–470.

Battaglia, M., Strambi, F. L., Bertella, S., Bajo, S., & Bellodi, L. (1999). First-cycle REM density in never depressed subjects with borderline personality disorder. Biological Psychiatry, 45, 1056–1058. doi:10.1016/S0006-3223(98)00213-3

Carlson, L. E., & Garland, S. N. (2005). Impact of mindfulness-based stress reduction (MBSR) on sleep, mood, stress, and fatigue symptoms in cancer outpatients. International Journal of Behavioral Medicine, 12, 278–285. doi:10.1207/s15327558ijbm1204_9

Chwastiak, L. A., & Von Korff, M. (2003). Disability in depression and back pain: Evaluation of the World Health Organization Disability Assessment Schedule (WHO DAS II) in a primary care setting. Journal of Clinical Epidemiology, 56, 507–514. doi:10.1016/S0895-4356(03)00051-9

Edinger, J. D., & Means, M. K. (2005). Cognitive-behavioral therapy for primary insomnia. Clinical Psychology Review, 25, 539–558. doi:10.1016/j.cpr.2005.04.003

Kessler, R. C., Berglund, P., Demler, O., Jin, R., Merikangas, K. R., & Walters, E. E. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 593–602. doi:10.1001/archpsyc.62.6.593

Kessler, R. C., & Merikangas, K. R. (2004). The National Comorbidity Survey Replication (NCS-R): Background and aims. International Journal of Methods in Psychiatric Research, 13, 60–68. doi:10.1002/mpr.166

Kessler, R. C., & Üstün, T. B. (2004). The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). International Journal of Methods in Psychiatric Research, 13, 93–121. doi:10.1002/mpr.168

Lenzenweger, M. F. (1999). Stability and change in personality disorder features: The longitudinal study of personality disorders. Archives of General Psychiatry, 56, 1009–1015. doi:10.1001/archpsyc.56.11.1009

Lenzenweger, M. F., Lane, M. C., Loranger, A., & Kessler, R. C. (2007). DSM-IV personality disorders in the National Comorbidity Survey Replication. Biological Psychiatry, 62, 553–564. doi:10.1016/j.biopsych.2006.09.019

Lenzenweger, M. F., Loranger, A. W., Korfine, L., & Neff, C. (1997). Detecting personality disorders in a nonclinical population: Application of a 2-stage procedure for case identification. Archives of General Psychiatry, 54, 345–351. doi:10.1001/archpsyc.1997.01830160073010

Linehan, M. M. (1993). Cognitive behavioral treatment for borderline personality disorder. New York, NY: Guilford Press.

Loranger, A. W. (1999). International Personality Disorder Examination: DSM-IV and ICD-10 interviews. Odessa, FL: Psychological Assessment Resources.

Loranger, A. W., Sartorius, N., Andreoli, A., Berger, P., Buchheim, P., Channabasavanna, S. M., . . .Regier, D. A. (1994). The International Personality Disorder Examination. Archives of General Psychiatry, 51, 215–224. doi:10.1001/archpsyc.1994.03950030051005

Muthén, B., & Muthén, L. (2008–2010). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén.

Roth, T., Jaeger, S., Jin, R., Kalsekar, A., Stang, P. E., & Kessler, R. C. (2006). Sleep problems, comorbid mental disorders, and role functioning in the National Comorbidity Survey Replication. Biological Psychiatry, 60, 1364–1371. doi:10.1016/j.biopsych.2006.05.039

Selby, E. A. (2013). Diagnostic efficiency of the International Personality Disorder Examination screening items for borderline personality disorder in a national epidemiologic sample. Manuscript in preparation.

Selby, E. A., & Joiner, T. E., Jr. (2009). Cascades of emotion: The emergence of borderline personality disorder from emotional and behavioral dysregulation. Review of General Psychology, 13, 219–229. doi:10.1037/a0015687

Selby, E. A., Ribeiro, J. D., & Joiner, T. E., Jr. (in press). What dreams may come: Emotional cascades and nightmares in borderline personality disorder. Dreaming.

Simon, G. E., & Von Korff, M. (1997). Prevalence, burden, and treatment of insomnia in primary care. American Journal of Psychiatry, 154, 1417–1423.

Sjöström, N., Wærn, M., & Hetta, J. (2007). Nightmares and sleep disturbances in relation to suicidality in suicide attempters. Sleep, 30, 91–95.

Wojnar, M., Ilgen, M. A., Wojnar, J., McCammon, R. J., Valenstein, M., & Brower, K. J. (2009). Sleep problems and suicidality in the National Comorbidity Survey Replication. Journal of Psychiatric Research, 43, 526–531. doi:10.1016/j.jpsychires.2008.07.006

Zohar, D., Tzischinsky, O., Epstein, R., & Lavie, P. (2005). The effects of sleep loss on medical residents’ emotional reactions to work events: A cognitive-energy model. Sleep, 28, 47–54.

Submitted: September 20, 2012 Revised: February 15, 2013 Accepted: April 29, 2013

Cognitive Behavioral Therapy for Adherence and Depression (CBT-AD) in HIV-Infected Injection Drug Users: A Randomized Controlled Trial

Acknowledgement: Funding for this project came from National Institute on Drug Abuse Grant R01DA018603 to Steven A. Safren. Clinical Trial Registration: Skills Based Counseling for Adherence and Depression in HIV+ Methadone Patients (NCT00218634; http://clinicaltrials.gov.offcampus.lib.washington.edu/ct2/show/NCT00218634). Portions of this article were presented at the fifth International Conference on HIV Treatment (IAPAC), May 2010, Miami, Florida.

Safren has served as a consultant for Dimangi and ISA Associates on technology-based adherence interventions and receives royalties from various book publishers, including Oxford University Press and Guilford Press. O'Cleirigh and Bullis have no disclosures to report. In the last 2 years, Otto has served as a consultant for MicroTransponder, Inc., and received research support from Shering Plough (Merck) as well as royalties from various book publishers. Pollack has served as a consultant for Eli Lilly, Medavante, Otsuka, and Targia Pharmaceuticals. In addition, he has received several grants from Bristol Myers Squibb, Euthymics, Forest Laboratories, and GlaxoSmithKline. Last, he has received equities from Medavante, Mensante Corporation, Mindsite, Targia Pharmaceutical Royalty/patent: SIGH-A, SAFER interviews. Stein has served as a consultant for Gilead, Bristol Myers Squibb, and visualmd.com.

We thank the many members of the study team who made this study a success. This includes (but is not limited to) Giselle Perez, Susan Michelson, Laura Reilly, Jessica Graham, Pamela Handelsman, Nicholas Perry, Nafisseh Souroudi, Jeffrey Gonzalez, Joseph Greer, Robert Knauz, Jonathan Lerner, Deb Herman, Luis Serpa, Jared Israel, Lauren McCarl, Allison Applebaum, Ellen Hendriksen, Christina Psaros, Lara Traeger, Sarah Markowitz, Carla Berg, the staff at BayCove, Community Substance Abuse Center, Habit OpCo, Roger Weiss, Kenneth Mayer, Robert Malow, and most importantly the study participants.

HIV continues to be a major public health concern in the United States, with no decline in rates of new infections and prevalence growing steadily (Centers for Disease Control and Prevention, 2008). The two most prevalent and interfering psychosocial comorbidities of HIV infection are clinical depression and substance use (Berger-Greenstein et al., 2007; Bing et al., 2001; Ruiz Perez et al., 2005). In a nationally representative probability sample of 2,864 adults who participated in the HIV Care Services and Utilization Study, 12-month prevalence rates for major depression, substance use without dependence (excluding marijuana use), and substance dependence were estimated as 36%, 25%, and 12.5%, respectively (Bing et al., 2001). Clinical depression and problematic substance use not only can cause significant distress and functional impairment but also can interfere with HIV treatment and care; both conditions have consistently been associated with poor antiretroviral therapy (ART) adherence (Catz, Kelly, Bogart, Benotsch, & McAuliffe, 2000; DiMatteo, Lepper, & Croghan, 2000; Lucas, Cheever, Chaisson, & Moore, 2001; Lucas, Gebo, Chaisson, & Moore, 2002; Paterson et al., 2000; Safren et al., 2001). A recent meta-analysis of 99 independent samples revealed a significant relationship between depression and adherence to HIV medications (Gonzalez, Batchelder, Psaros, & Safren, 2011).

Individuals with injection drug use (IDU) histories continue to compose a large proportion of individuals living with HIV in the United States (Centers for Disease Control and Prevention, 2010). The most recent estimates available from 37 states with confidential name-based HIV-infection reporting suggest that approximately 12% of men and 15% of women living with HIV in 2008 acquired HIV through IDU (Centers for Disease Control and Prevention, 2010). However, these estimates only account for HIV-infection directly attributable to IDU and thus do not reflect the secondary impact of transmission through sexual contact with a partner who acquired HIV through IDU.

In addition to the importance of adherence for self-care and optimization of the benefits of ART, adherence may be important in the transmissibility of HIV. HIV transmission is highly dependent on the amount of HIV viral load present in any given individual's blood and genital secretions (Hull & Montaner, 2011). HIV viral load can be reduced to an undetectable level through successful antiretroviral therapy, which seems to significantly reduce transmission risk between HIV-serodiscordant partners (Attia, Egger, Muller, Zwahlen, & Low, 2009). Accordingly, increased adherence to ART among opioid-dependent individuals living with HIV may also provide a secondary public health benefit of contributing to HIV prevention efforts (Hull & Montaner, 2011) and is a part of new emerging “test, treat, and retain” strategies.

Injection drug users living with HIV face multiple changes to successful HIV treatment, including a poorer virological response to ART compared with other HIV-infected populations, poor adherence to ART, and increased rates of attrition during interventions (Keiser et al., 2012; Weber et al., 2009). Research suggests that HIV-infected individuals currently receiving treatment for IDU or opioid dependence continue to struggle with adherence to ART (Weber et al., 2009). Avants, Margolin, Warburton, Hawkins, and Shi (2001), for example, found that more than a third of HIV-positive patients receiving methadone maintenance treatment reported less than 80% adherence to their HIV medication regimens, a rate that potentially increases the risk of developing a drug-resistant strain of the virus. Even when ART doses were directly administered and supervised in a methadone clinic-based program, continued substance use was associated with an increased risk of nonadherence and intervention dropout (Lucas et al., 2007).

HIV-positive individuals are also at an increased risk for major depression, with prevalence rates suggesting that twice as many HIV-positive individuals suffer from depression than demographically matched HIV-negative individuals (Ciesla & Roberts, 2001). Among a sample of triply diagnosed patients with HIV, substance abuse, and psychiatric illness, Berger-Greenstein et al. (2007) reported that over 70% of participants met criteria for major depression; self-reported depressive symptoms were also significantly related to worse HIV medication adherence and lower CD4 cell count. A prospective observational study by Riera et al. (2002) reported that among 202 HIV-positive patients, depression and methadone maintenance treatment were independent predictors of poor adherence to antiretroviral medications. During an evaluation of depressive symptoms and symptomatic response among HIV-infected injection drug users who were enrolled in a randomized controlled trial of directly observed ART, improvements in depression over 6 months were associated with increases in CD4 cell count and adherence, whereas worsening in depression was associated with active drug use and increases in plasma viral RNA levels (Attia et al., 2009; Hull & Montaner, 2011; Springer, Chen, & Altice, 2009).

Maintaining excellent adherence can be a difficult and challenging process for a sizeable proportion of individuals living with HIV. Poor adherence decreases the benefits of ART, as well as chances of prolonged survival (e.g., García de Olalla et al., 2002; Thompson et al., 2010). To address these barriers, there is an emerging evidence base for the efficacy of interventions for ART adherence (Amico, Harman, & Johnson, 2006; Simoni, Amico, Pearson, & Malow, 2008; Simoni, Frick, Pantalone, & Turner, 2003; Simoni, Pearson, Pantalone, Marks, & Crepaz, 2006). However, to date, most interventions have produced only modest effects, have focused directly on adherence, and have not addressed psychosocial comorbidities that may moderate the degree to which the interventions would be successful. When one considers the symptoms of a depressive episode (e.g., persistent sad mood, loss of interest, concentration problems, low energy, feelings of excessive worthlessness/guilt), it is not difficult to see how these symptoms could interfere with the acquisition or use of skills necessary to improve adherence and could potentially minimize the efficacy of adherence training interventions that do not directly treat depression.

Our prior work involved developing (Safren et al., 2004) and initially testing (Safren et al., 2009) cognitive behavioral therapy for adherence and depression (CBT-AD) in HIV. We showed in a crossover design that integrating adherence counseling using our Life-Steps protocol (Safren et al., 2001) with CBT for depression was successful at both increasing adherence and reducing depression in individuals with HIV and depression (Safren et al., 2009). Although individuals with active substance use were excluded from this initial study, HIV infection due to IDU accounts for 18.5% of cases of HIV among adults in the United States (Centers for Disease Control and Prevention, 2008, 2010). As articulated above, these individuals may be particularly at risk for depression due to the multiple stressors involved with managing comorbid HIV-infection and opioid dependence. Triply diagnosed individuals—those with HIV, clinical depression, and substance use (e.g., opioid abuse/dependence)—represent a population uniquely at risk for nonadherence. Accordingly, in designing this trial, a priori, we sought to examine the degree to which intervening on depression would assist the ability to benefit from evidenced-based adherence counseling.

The primary objective of the current study was to test, in a randomized controlled trial, CBT-AD in patients with HIV, depression, and opioid dependence who were undergoing treatment for their substance use disorder. We hypothesized that those who were assigned to the CBT-AD condition would have better adherence (primary outcome), decreased depression, and improved biological outcomes (e.g., decreased viral load and increased CD4+ lymphocyte counts) than the comparison group (enhanced treatment as usual [ETAU]) and that these gains would be maintained over the 9-month follow-up period.

Study Subjects and Setting

Enrollment occurred between July of 2005 and October of 2008 and included (89 randomized) individuals between the ages of 18 and 65 years who were HIV-seropositive, prescribed antiretroviral therapy for HIV, endorsed a history of injection drug use, were currently enrolled in opioid treatment for at least 1 month, and met criteria for a diagnosis of current or subsyndromal depressive mood disorder (72 major depressive disorder; one dysthymia; 16 bipolar disorder, most recent episode depressed). Subsyndromal depression (n = 10) was defined as a past history of major depression, with a current level of residual symptoms (Clinical Global Impression [CGI—see Measures section] of at least 2) that did not meet diagnostic threshold (i.e., due to antidepressant therapy).

Treatment for opioid dependence varied, with the majority of participants having received methadone (70%, n = 63) and the remainder receiving suboxone therapy (5.6%, n = 5), group (4.5%, n = 4) or individual substance abuse counseling (7.9%, n = 7), active participation in Narcotics Anonymous (4.5%, n = 4), or other active substance abuse treatment (6.7%, n = 6). There were no significant differences in type of substance abuse treatment between the experimental and control conditions.

Excluded individuals were those with any active untreated or unstable major mental illness that would interfere with study participation (e.g., active mania or psychosis), inability or unwillingness to provide informed consent, or current participation in cognitive behavioral therapy (CBT) for depression.

Participant demographics are depicted in Table 1, and raw study-related outcomes, including baseline, appear in Table 2 (note that analyses and graphs used general linear modeling [GLM] and hierarchical linear modeling [HLM] adjusted scores). The study sample was of a predominately lower socioeconomic status, with only 4% working or being in school full-time and 67% on disability. There were no differences on baseline demographic variables across conditions. There was a baseline difference for CD4 count, with the CBT-AD group having a higher CD4 count than the comparison condition, t(87) = 2.76), p < .01—see Table 2. Hence, baseline levels were covaried in longitudinal analyses.

Sociodemographic Characteristics of Participants

Unadjusted Mean Descriptive Scores for Outcomes Across Conditions and Time

The sample had substantial psychosocial comorbidity, with 62% having at least one additional Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM–IV; American Psychiatric Association, 1994) diagnosis besides depression and substance abuse disorder (see Table 2). Sixty-five percent of the randomized sample had recent illicit substance use as assessed by a combination of toxicology screening and self-report at baseline. During the clinician-administered assessments administered at baseline, participants were asked to report any substance use over the past 30 days. Approximately one third (30.3%) of randomized participants reported polysubstance use during the past 30 days, 23.6% reported alcohol use; 5.6% reported alcohol use to intoxication; 25.8% reported heroin use; 75.3% reported methadone use; 23.6% reported either opiate or analgesic use; 37.5% reported either sedative, hypnotic, or tranquilizer use; 25.8% reported cocaine use; 16% reported cannabis use; and 1.1% reported hallucinogen and inhalant use over the past 30 days; there was no reported amphetamine or barbiturate use. In addition to the clinician-administered assessment of substance use over the past 30 days, participants also provided a saliva sample for a toxicology screen. Of the 89 participants randomized, 77.2% tested positive for methadone use, 22.5% for cocaine use, 12.7% for opiate use, 8.8% for benzodiazepine use, 5% for cannabis use, and 1.3% for amphetamine and barbiturate use. There were no significant differences in reported substance use or toxicology results based on randomization condition.

For the first participants (n = 40), all study visits took place at one of four methadone clinics in the greater Boston area. Recruitment was later expanded for two major reasons: (a) we discovered that some potential participants were not comfortable referring themselves or being seen for the study at the methadone clinics despite measures to protect confidentiality about being in an HIV study, and (b) during the time of the study, more options became available for treatment of opioid dependence, such as suboxone. Seventy percent were on methadone at baseline, 6% suboxone, 7% Narcotics Anonymous/Alcoholics Anonymous only, and 18% counseling (individual or group); 56% were on an antidepressant medication at study entry Hence, participants were then recruited through community outreach and HIV clinics at Massachusetts General Hospital (MGH; n = 8) and Rhode Island Hospital (n = 9). The remaining participants (n = 32) were referred by other study participants or through community outreach and recruitment flyers posted in other HIV care or substance abuse (including additional methadone clinic) settings but were seen at an MGH-based research clinic. This adaptive trial design allowed us to keep up with the ever-changing epidemic as the trial was in process.

After a complete description of the study was provided to the participants, study clinicians obtained written informed consent. All study procedures were approved by the Institutional Review Boards at Massachusetts General Hospital (MGH) in Boston, MA and at Rhode Island Hospital in Providence, RI.

Study Design and Procedures

Study visits

After an initial evaluation to determine study eligibility and a 2-week period during which participants started using the electronic pill caps, there were four major study assessment visits: T1 was the baseline assessment; T2 was the post-treatment outcome, which happened at end of intervention for those in the experimental arm (approximately 3 months after baseline for participants in both arms); T3 was a 3-month follow-up (occurring 6 months from baseline); and T4 was a 9-month follow-up (occurring 12 months from baseline). These assessment points were chosen so that we could have four major outcome assessments over the course of 1 year of study involvement, allowing for longitudinal analyses for the follow-up assessments. The post-treatment assessment was at approximately 3 months, and that was selected to allow participants in the CBT-AD condition enough time to come to nine treatment sessions, accounting for issues such as snow and life events that might not allow for coming consistently every week. The 3-month (post-treatment) assessment was to examine acute outcomes during the treatment, right after it was discontinued. The T3 and T4 outcomes were to examine short-term and long-term maintenance of gains, as well as have time to see adherence or depression-based changes in any biological outcomes.

These assessments included electronic pill cap evaluations (Medication Event Monitoring System [MEMS], manufactured by AARDEX) for adherence, assessment of depression by self-report and an independent assessor blinded to study condition, as well as HIV plasma RNA and CD4+ lymphocyte counts either drawn for the study or abstracted from participants' medical records if collected in the month prior to the assessment. Samples acquired during the baseline assessment with a viral load of over 1,000 copies per milliliter were tested for genotypic resistance. Participants received $50.00 for the major assessments (e.g., baseline and three follow-up visits) and $25.00 for the weekly visits during the acute study period.

Randomization

Study coordinators randomly assigned participants at their first visit after the baseline in blocks of two, stratified by biological sex, depression severity (current major depression or residual symptoms only), and adherence (baseline MEMS-based adherence above or below 80%). Assignment to study condition (CBT-AD or ETAU) was concealed from both study therapists and participants until the conclusion of the first counseling visit (see below).

Assessment Measures

Primary outcome: Adherence

MEMS caps recorded each instance of bottle opening, monitoring the antiretroviral medication that the participants considered the most difficult to remember or the dose taken most frequently. To account for doses that participants may have taken without opening the pill cap (e.g., took out afternoon doses when they opened the pill bottle in the morning), we counted a dose as taken if participants could recall specific instances when they took their medications but did not use the cap (Liu et al., 2001, 2006; Llabre et al., 2006). A dose was considered missed if it was not taken within a 2-hr window of the designated time. If participants were using a pill-box prior to entering the study, we encouraged them to monitor a pill that is taken concurrently with another, with one going in the pill box and the other going in the bottle, so that the function of the pill-box (i.e., knowing if a pill has been taken/organization) could be maintained. In both conditions, if there were discrepancies between self-report and MEMs data, research assistants (RAs) or therapists would interview participants further to determine whether their cap should be replaced and/or try to figure out what may have caused this discrepancy.

For the acute outcome (baseline to post-treatment), adherence was operationalized as the percentage of MEMS-based adherence since the last visit; visits were scheduled weekly or, at most, every 2 weeks. For the follow-up longitudinal analyses, we used adherence in the past 2 weeks. This is consistent with our prior studies that used MEMs, and balances having an adequate sampling of time with not overlapping too much with the time that the participants were still potentially improving from the intervention (Safren, Hendriksen, DeSousa, Boswell, & Mayer, 2003; Safren et al., 2004; Safren et al., 2009).

Clinician-Administered Assessments

Enrollment visit

The initial evaluation to establish study eligibility included a diagnostic evaluation of DSM–IV diagnoses using the Mini International Neuropsychiatric Interview (MINI; Sheehan et al., 1998), one of the most widely used diagnostic assessments. This evaluation was completed by one of the study therapists and was presented for review and diagnostic consensus by the study team.

Independent assessments

An independent assessor (IA), who remained blind to study condition, conducted the clinician-administered outcome assessments. The IA visits included administration of (a) the Montgomery-Asberg Depression Rating Scale (MADRS; Montgomery & Asberg, 1979), and (b) a rating of global distress and impairment for depression and substance abuse using the Clinical Global Impression (CGI; National Institute of Mental Health, 1985) for severity (e.g., 1 = not ill to 7 = extremely ill). The MADRS and CGI scores were regularly reviewed through audiotape supervision with another blinded assessor. Matching the number of study visits between the two conditions helped preserve the blinding (e.g., if the IA saw the participant in the waiting room frequently, the participant could be in either condition). Additionally, participants were reminded before and during the IA visits not tell the assessor which study condition they were in.

Participant Measure of Depression

Participants completed the self-reported Beck Depression Inventory—Short Form (BDI-SF; Beck & Beck, 1972) during each visit. This measure was designed for use with medical populations, removing many of the somatic symptoms of depression that might be confounded with medication side-effects or physical functioning.

Biological Outcome Measures

At the major study assessment visits, participants who did not have an HIV plasma RNA or CD4+ lymphocyte test in the prior month accessible through clinic chart review provided blood for testing.

Intervention Conditions

Participants in both the treatment (CBT-AD) and comparison (ETAU) conditions received a single-session intervention on HIV medication adherence (Life-Steps), which involved 11 informational, problem-solving, and cognitive behavioral steps (Safren, Otto, & Worth, 1999). In each step, participants and the clinician define the problem, generate alternative solutions, make decisions about the solutions, and develop a plan for implementing them. Participants also received adherence tools such as assistance with a schedule and a cue-dosing watch that could sound two alarms per day. To enhance treatment as usual, they also had a letter mailed to their medical providers documenting the participant's depression or other psychiatric disorders and suggesting that these conditions should continue to be assessed or treated.

In addition to this, those assigned to the experimental condition also received eight sessions of CBT-AD (Safren, Gonzalez, & Soroudi, 2007a, 2007b). Accordingly, this was nine sessions total, with Life-Steps being Session 1, followed by eight CBT-AD sessions. This approach integrated continued adherence counseling with traditional CBT techniques for the treatment of depression. Module 1 (≈1 session; average in this study = 1.0 session) and provided psychoeducation about HIV and depression and a motivational interviewing (MI) exercise designed to set the stage for behavioral change. The MI exercise involved examining the pros and cons of changing and not changing self-care behaviors, as well as a discussion of a metaphor about treatment. Module 2 (≈1 session; average in this study = 1.2 sessions across participants) focused on behavioral activation and activity scheduling, which was designed to increase regularly occurring activities that involve pleasure and mastery. Module 3 (≈3 sessions; average in this study = 2.4 sessions across participants), cognitive restructuring, involved training in adaptive thinking, such as identifying and restructuring negative automatic thoughts. Module 4 (≈2 sessions; average in this study = 1.0 sessions across participants), problem-solving, involved training in selecting an action plan for problems and breaking this plan into manageable steps (Nezu, Nezu, Felgoise, McClure, & Houts, 2003). Module 5 (≈1 session; average in this study = 1.0 session across participants), relaxation, involved training in progressive muscle relaxation and diaphragmatic breathing. Some participants had a review session as needed (average = 0.4 sessions across participants). Sessions were approximately 50 min long and occurred weekly, with the goal of completion in approximately 3 months. A more detailed description of the intervention can be found in our published manuals (Safren et al., 2007a, 2007b). Flexibility in the number of sessions devoted to any module was permitted to address the complexity and variability of issues facing participants with HIV, depression, and intravenous drug use histories.

Study interventionists included clinical psychologists, psychology pre- and post-doctoral fellows in clinical psychology, and one master's-level psychologist. Training involved didactic learning from the modules and supervision using audio-recordings of sessions. To maximize therapist adherence to the intervention, all sessions were audio-recorded for monitoring and supervision. Interventionists met with a clinical supervisor on a weekly basis for clinical supervision where cases and interventions were discussed. For new interventionists, all sessions were listened to by the clinical supervisor, for at least the clinician's first participant, for feedback purposes. On an ongoing basis, one therapist per week was assigned an audiotape (of a different therapist's session) to review and complete a checklist for interventionist adherence, including whether the specific components of the modules of treatment were, in fact, delivered. Traditional monitoring of treatment fidelity, as typically done in randomized controlled trials of psychological treatments with circumscribed samples (usually individuals meeting criteria for one psychological disorder and receiving treatment for that disorder), was not possible for this study population of triply diagnosed individuals, because urgent life events often occurred and were addressed in treatment. Hence, the supervision sessions and fidelity ratings were used to develop a process for future research. This balanced the need for therapists to adhere to the general principals of CBT and intervene with respect to self-care behaviors, while being flexible regarding the order of the manualized modules, fitting the manualized modules to the clients' needs, and providing CBT strategies to assist with depression and self-care, regardless of what particular chapter would have been next in the treatment protocol. Accordingly, by the end of this process, approximately 5% of sessions were rated using the system that was developed during the study. A future article will more fully detail these data, and a current trial that seeks to examine active components of treatment (e.g., that compares this intervention to a credible, time-matched, active control treatment with both fidelity and contamination ratings) is currently underway.

After the Life-Steps adherence counseling session, participants in the ETAU condition also had eight study visits before the post-treatment assessment (to make the nine sessions total). During these visits, participants had their MEMS cap downloaded, completed the BDI-SF, and received the same number and timing of visits as those in the CBT-AD condition.

Statistical Analyses

Our prior study had an effect size of 1.0 for the primary MEMS-based adherence intent to treat outcome (Safren et al., 2009). That study, however, did not include those with comorbid substance dependence and hence we powered the study for an effect size of d = 0.8, which yielded a goal of approximately 100 participants using analysis of variance for MEMS-based adherence. Longitudinal modeling, however, as described next, allows for greater power using all available data. The actual sample size included 89 randomized participants.

HLMs (with HLM 6.06 software) were used to evaluate acute study outcomes when there were at least three data points (Raudenbush, Bryk, Cheong, & Congdon, 2004). This included MEMS-based adherence during the pre-post treatment phase, which was collected at each study visit; self-reported depression during the pre-post treatment phase, which was also collected at each study visit; and follow-up analyses for all study outcomes.

Repeated measures (GLMs) were used for pre-post assessments of depression by independent assessors (e.g., MADRS and CGI) and pre-post biological markers of HIV disease (e.g., plasma RNA viral copies and CD4 cell count), as there were only two assessment time points, and HLM or other mixed effects modeling could not be used. In these analyses, study condition assignment was the between-subjects factor. This was part of the a priori analysis plan: to first examine pre-post outcomes and second examine maintenance of any gains and biological endpoints over the longitudinal follow-up.

For pre-post HLM analyses where we had repeated measures (MEMS-based adherence and BDI), the Level 1 HLM model included the time variable (weeks since baseline), which provided the structure of the model for the outcome variable of interest. The Level 2 model tested the significance of the treatment effect and is estimated from the significance of the slope (gamma coefficient) associated with the random assignment variable (CBT-AD or ETAU). As there were significant differences between the randomly assigned conditions on CD4 cell number at study entry baseline, CD4 cell number was controlled in each of the main outcomes analyses (see ^{footnote 1}).

For maintenance of treatment effects HLM analyses, to model the slope in study outcomes across the follow-up assessments using baseline values, the Level 1 model included time, and the Level 2 model tested study condition, controlling for pre-randomization levels. Treatment effects on log viral load and CD4 cell count change across the follow-up time period were estimated by controlling for resistance to at least one antiretroviral medication in the Level 2 model.

For the HLM models, all continuous measures in the Level 2 model were centered about their group means, and all dichotomous variables were coded 1/0. Model parameters were estimated using full maximum-likelihood estimation with robust standard errors. In all analyses that used HLM, unconstrained models were run to confirm significant individual variation about the slope and intercept before accounting for random assignment. For all analyses, the Type I error rate adopted was a p of .05. Effect sizes for HLM growth estimates were calculated for statistically significant outcomes using the formula dGMA-raw = γ11(time)/SDraw, in line with current recommendations for communicating effect magnitude (Feingold, 2009; Raudenbush & Liu, 2001).

For all analyses involving comparing the two study arms, analyses presented controlled for baseline CD4 cell count differences between groups (there were no other baseline differences between the two study conditions). When the same analyses were conducted not controlling for baseline CD4 cell count differences between groups, the pattern of results were the same (see ^{footnote 1}).

Participant Characteristics

Participant flow throughout the duration of the study is depicted in Figure 1. Ninety-one percent of those randomized were retained for the acute outcome assessment (n = 86), and 84% (n = 79) returned to at least one post-treatment follow-up and hence could be used for follow-up analyses. Although there was a raw number greater loss to attrition at the final assessment point in the ETAU condition (eight out of 44 participants), compared with the CBT-AD condition (15 out of 45 participants), this difference was not statistically significant. Raw scores for study outcomes are presented in Table 2 by study condition (note that HLM and GLM analyses use adjusted scores). At baseline, 11.2% of the sample collapsed across condition had genotypic resistance. There were no study-related adverse events that occurred during the study.

Figure 1. Consolidated Standards of Reporting Trials (CONSORT) participant flow chart. IDU = injection drug use; CBT-AD = cognitive behavioral therapy for adherence and depression; ETAU = enhanced treatment as usual.

Baseline to Post-Treatment Outcomes

Adherence (MEMS)

There was a significant upward slope in MEMS-based adherence, γslope = 0.47, t(88) = 2.16, p = .033, during the treatment period, indicating improved adherence for the study participants as a whole. In addition, there was significant individual variation about the slope, ρslope = 2.35, df(87), χ² = 203.01, p < .001, providing the justification for conducting the analysis by randomized group (Level 2 analysis). When adding treatment condition, and, as a covariate, baseline CD4, to the model, the increase in adherence was significantly greater over time in the CBT-AD condition (11.8 percentage points) than in the comparison condition (0.5 percentage points), γslope = 0.887, t(86) = 2.38, p = .02, dGMA-raw = 0.64 (see Figure 2).

Figure 2. Pre-post outcomes: Longitudinal (hierarchical linear modeling) analysis of MEMS-based adherence and depression (BDI-SF). CBT-AD = cognitive behavioral therapy for adherence and depression; ETAU = enhanced treatment as usual; MEMS = Medication Event Monitoring System; BDI-SF = Beck Depression Inventory—Short Form (Beck & Beck, 1972). Data points are adjusted scores using MEMS-based adherence and BDI-SF scores for the time since prior visit.

Depression rated by independent assessor

There were significantly greater reductions in depression for the CBT-AD condition relative to the comparison condition for both the MADRS, F(1, 78) = 9.72, p = < .01, d = 0.55, and CGI, F(1, 78) = 17.14, p < .001, d = 0.85 (see Table 2). These analyses used baseline CD4 as a covariate.

Self-reported depression (BDI-SF)

There was a significant decreasing slope, γslope = –0.23, t(88) = –3.52, p < .01, for self-reported depression over time. When accounting for treatment condition, and controlling for baseline CD4 as a covariate, those in the CBT-AD condition experienced a significant estimated reduction in depression symptoms (5.1 points on the BDI-SF) compared with a nonsignificant change in the control condition (<1 point on the BDI-SF), γslope = –0.320, t(86) = –2.39, p = .02, dGMA-raw = 0.63.

Biological outcomes

Although treatment effects on HIV viral load and CD4 count were more likely to emerge only at follow-up, acute effects (e.g., baseline to post-treatment) were examined and are presented in Table 2 for consistency. Controlling for medication resistance, randomized effects were not significant for either CD4 cells, F(1, 74) = 1.92, p = .32, d = 0.10, or for log viral load, F(1, 71) = 1.08, p = .30, d = 0.20, with the additional control for baseline CD4.

Post-Intervention Follow-Up Assessments at 3 and 9 Months

Adherence (MEMS)

MEMS-based adherence gains acquired across CBT-AD treatment were not maintained during follow-up as evidenced by the significant downward slope for MEMS-based adherence, γslope = –0.294, t(79) = –3.24, p < .01, that did not differ by study condition, γslope = 0.20, t(76) = 1.23, p = .22, dGMA-raw = 0.21 (see Figure 3). Baseline CD4 differences were controlled for as a covariate in these analyses.

Figure 3. Follow-up outcomes: Analysis of MEMS-based adherence and depression (BDI-SF). CBT-AD = cognitive behavioral therapy for adherence and depression; ETAU = enhanced treatment as usual; MEMS = Medication Event Monitoring System; BDI-SF = Beck Depression Inventory—Short Form (Beck & Beck, 1972); F/U = follow-up. Follow-up outcomes used hierarchical linear modeling adjusted scores for prior 2 weeks MEMS-based adherence.

Clinician-assessed depression (MADRS and CGI)

Depression gains as assessed by blinded clinicians were maintained, which was demonstrated on the CGI by a trend for continued improvement during the follow-up time period, γslope = –0.008, t(80) = –1.93, p = .06, with no differential improvement by condition over follow-up after acute treatment ended, γslope = 0.011, t(78) = 1.59, p = .12, dGMA-raw = 0.51. Similarly, the MADRS also demonstrated a trend for a continued improvement for the group as a whole, γslope = –0.052, t(80) = –1.69, p = .09, with no differential improvement by condition, γslope = 0.078, t(78) = 1.47, p = .14, dGMA-raw = 0.61. These analyses had baseline depression scores and baseline CD4 as covariates.

Self-reported depression (BDI-SF)

Improvements in depression acquired during the CBT-AD intervention and assessed via self-report were maintained during follow-up, as evidenced by no significant change in self-reported depressive symptoms for the whole sample, γslope = 0.03, t(73) = –1.27, p = .21, or by group assignment when controlling for baseline BDI and CD4 differences, γslope = –0.01, t(72) = –0.30, p = .76, dGMA-raw = 0.13.

Biological outcomes (HIV viral load, CD4 cell count)

There was no significant change in log viral load over the follow-up time period for the group as a whole, γslope = –0.0015, t(75) = –0.40, p = .69, or based on group assignment, γslope = 7.28 × 10^−x4, t(74) = –0.165, p = .87, dGMA-raw = 0.02, controlling for baseline log viral load, CD4 cell number, and resistance. There were also no differences across the two conditions in the percentage of participants who attained a suppressed viral load, γslope = 5.0 × 10^−x5, t(74) = –0.002, p = .98, dGMA-raw < 0.01.

Over the follow-up period, the slope of CD4 cell number was nonsignificant for the entire sample, γslope = 0.590, t(79) = 1.08, p = .29. When condition was added to the model, there was a significant increase in CD4 cells in the CBT-AD condition compared with the control condition, controlling for baseline CD4 and medication resistance, 61.2 CD4 cell increase versus 22.4 CD4 cell decrease, γslope = 2.09, t(76) = 2.20, p = .03, dGMA-raw = 0.60. Note these differences were also significant without controlling for baseline CD4 and medication resistance.

The current study examined the use of a time-limited intervention (CBT-AD) addressing both adherence and clinical depression in a sample of triply diagnosed individuals with HIV. The intervention had acute and significant effects on both adherence and depression during the time in which the intervention was being delivered. MEMS-based adherence in the CBT-AD group improved approximately 11.8% from baseline and 11.3% over the comparison condition during treatment. The magnitude of this effect is potentially clinically significant in that it has been suggested that a 10% change in adherence can result in improved HIV outcomes (Bangsberg, 2006; Liu et al., 2001). However, after the intervention ended, MEMS-based adherence decreased, whereas intervention-related improvements in depression remained relatively stable. By way of contrast, our prior study of CBT-AD with depressed HIV-infected participants who were not also struggling with IDU histories and substance dependence showed sustained effects on both adherence and depression, and improvements in viral load over time (Safren et al., 2009). As such, it appears that our intervention was not resilient to the psychosocial challenges magnified by the context of opioid dependence, HIV, and depression. It is not clear which aspect or aspects of substance dependence (e.g., lapses in substance use, neuropsychological impairment, psychosocial stress, schedule disruptions, altered motivation, altered meaning of pill taking) may have contributed to this loss of efficacy for the adherence but not depression outcomes, and moderators will be explored in future secondary articles. Our findings indicate that, for triply diagnosed individuals, continued adherence counseling may be necessary to maintain or potentially augment adherence gains, even when depression symptoms improve.

With respect to the depression findings, the average depression score on the MADRS for those in the CBT-AD conditions at baseline was in the range for “moderate” severity and only “mild” at the 12-month follow-up assessment. This 40% decrease in symptoms at post-treatment, which was maintained at 12 months, represents a clinically meaningful reduction (Müller, Himmerich, Kienzle, & Szegedi, 2003; Robertson, 1983).

Differences in viral load did not emerge across the two conditions over time; however, there were differential improvements in CD4 in those who received CBT-AD versus those who received ETAU when statistically controlling for baseline group differences. Although preliminary, this finding is consistent with the results of other HIV-related psychosocial interventions demonstrating that psychosocial interventions can directly improve biological indicators of HIV pathogenesis, including CD4 cells counts (Petrie, Fontanilla, Thomas, Booth, & Pennebaker, 2004) and HIV viral load (Antoni et al., 2006). This finding, that there were differences in depression and CD4 but not long-term differences in adherence or viral load is noteworthy but certainly requires replication. Despite the absence of viral load change, it is possible that our participants still reaped some psychoneuroimmunological benefit due to sustained reductions in depression as evidenced by an increase in CD4 lymphocyte count over time, which occurred after covarying out baseline differences. The reduction of the immunosuppressive effects of depression-related dysregulation of the catecholaminergic (norepinephrine) or HPA (cortisol) axis may support sustained increases in CD4 cell counts (Leserman, 2003). In fact, Antoni et al. (2006) reported that treatment-related decreases in HIV viral load were mediated by reductions in depressed mood. The preliminary finding that treating depression is associated with improved immunity is also consistent with noninterventional studies showing a relationship between depressive symptoms and change in CD4 cells over time (Ickovics et al., 2001; Ironson et al., 2005; Leserman, 2008). Future secondary analyses will examine mediational or other potential pathways with these data. This immunological finding requires replication, as it is possible that there are other explanations for these results, such as regression to the mean after different values at baseline. Additional randomized controlled trials of interventions that improve depression on immunologic outcomes are required to examine this further. With respect to the absence of viral load findings, it is possible that the ability to detect viral load differences was limited because of the relatively low average viral load levels at baseline.

It is noteworthy that prior studies have found short-term ART adherence gains simply as a result of using a MEMS cap for monitoring without additional intervention, gains sometimes lasting up to 40 days (Deschamps, Van Wijngaerden, Denhaerynck, De Geest, & Vandamme, 2006; Wagner & Ghosh-Dastidar, 2002); although our post-treatment outcome assessment target time window was 90 days. Accordingly, it is possible that the first set of MEMS-based adherence assessments (at baseline—first 2 weeks) represent an improvement over what their true baseline adherence would have been. Hence, decreases in MEMS-based adherence over time to the final follow-up may be due to the waning effects of adherence monitoring with MEMS on adherence over time. The randomized design with the post-treatment outcome being approximately 3 months after randomization (i.e., more than 40 days) showed differences between the intervention and comparison conditions at post-treatment, indicating that it is unlikely that there would be a difference in a sensitization effect to MEMS-based adherence across the two study conditions.

There were eight out of 44 individuals lost to follow-up in the CBT-AD condition and 15 out of 45 from the ETAU condition. Although this was not statistically significant, it raises an issue that may be common to adherence interventions. More specifically, if depression is associated with adherence, and those who are adherent to study participation are more likely to also be adherent to HIV medications, it is possible that attrition is associated with poorer adherence and more severe depression. In this scenario, the actual treatment effect would be greater than what was observed, because the control participants who dropped out would be those with worse depression and adherence. Conversely, it is also possible that control participants who dropped out did so because they did not perceive benefit from participation in the study. Our clinical experience meeting with these participants, however, suggests that that the latter was not the case.

There are several additional limitations to the present study that should be noted. First, although MEMS are an objective indicator of adherence, it is possible that adherence was underestimated if the MEMS cap was not used. Accordingly, we asked participants at each assessment whether they recalled taking pills without using the cap and used a corrected adherence score (Liu et al., 2001, 2006; Llabre et al., 2006). Second, as part of a program of research to test a treatment, and to examine whether treating depression is necessary to benefit from an adherence intervention before trying to dismantle the “active ingredients,” the comparison group was not attention-matched. Although CBT-AD and ETAU participants came for the same number of visits and had the same incentive payments, we do not know if the positive acute adherence outcomes are related to specific elements of CBT-AD beyond the Life-Steps adherence counseling. Third, despite procedures to ensure confidentiality, we discovered that individuals in methadone treatment centers may have been reluctant to refer themselves to the study because of the fear of having their HIV status “outed.” Recruitment efforts that were expanded to address this barrier threatened internal validity, but this concern was balanced by achieving greater participant heterogeneity and therefore generalizability. Fourth, the participant incentives may have driven participation, and generalization of findings to opioid-stabilized patients who are less likely to come for such counseling may be limited. Fifth, we had hoped for a sample size of 100, and only 89 could be randomized because of the complexities of recruitment. Finally, given the complexity of the population, monitoring fidelity to the intervention was a work in progress throughout the study. Accordingly, by the later half of the study, we had developed a fidelity monitoring checklist that allowed for therapist flexibility when emergent life events would occur, with the ability to change the order of modules and/or utilize the treatment module most relevant to an emergent concern when it would occur. Although this can be seen as a threat to internal validity, it conversely increases the external validity of the intervention, as this is what clinicians would likely do in real-world clinical practice.

Future research should examine the cost-effectiveness of the intervention as part of a larger effectiveness trial involving substance abuse treatment programs that have high numbers of HIV-infected patients in their practice base. Accordingly, if substance abuse counselors or less trained interventionists could integrate this intervention into their current counseling, there would not be incremental costs, and adherence counseling could be maintained, potentially allowing for sustained benefits.

The findings, however, suggest that CBT-AD is a potentially useful strategy for increasing adherence and decreasing depression in HIV-positive patients with a history of IDU who are in substance abuse treatment. Adherence gains were only present during the time that the treatment was ongoing, suggesting that booster, additional, or continued adherence counseling sessions may be needed to sustain improvements in adherence outcomes in this population of individuals struggling with multiple comorbidities. It appears that this intervention, integrated into substance use counseling in methadone or other drug treatment programs, would be beneficial for individuals with HIV and substance abuse disorders. For depression, this intervention resulted in sustained improvement, and CD4 cell counts increased in the CBT-AD condition compared with the control condition over time. Whereas prior studies have found correlational associations of depression to CD4, this is the first study to suggest a possible effect when depression was successfully treated.

¹ We thank the anonymous reviewers for suggesting that we control for CD4 in all outcome analyses, as CD4 differed in magnitude between study arms. Analyses were initially conducted without controlling for baseline CD4, and the pattern of results was the same when controlling for and when not controlling for CD4 in these analyses.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.